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Report of the Auditor General 

to the House of Commons 

for December 2000 


Foreword 


I am pleased to present the December volume of my 2000 Report. This Foreword is followed by “Matters of 
Special Importance — 2000” and the Main Points from all of this year’s chapters. This volume of my Report 
contains the following 18 chapters: 


18. Governance of Crown Corporations 

19, Reporting Performance to Parliament: Progress Too Slow 

20. Managing Departments for Results and Managing Horizontal Issues for Results 

Bis Post-Secondary Recruitment Program of the Federal Public Service 

ae: Development of the Universal Classification Standard: A Follow-up 

23. Information Technology: Acquisition of Goods and Services 

24. Federal Health and Safety Regulatory Programs 

as Canadian Food Inspection Agency — Food Inspection Programs 

26. Health Canada — Regulatory Regime of Biologics 

Pil: Canadian Nuclear Safety Commission — Power Reactor Regulation 

28. Follow-up of Previous Recommendations on Health and Safety Regulatory Programs 

29. Treasury Board Secretariat - The Government’s Annual Contracting Activity Report — 1998 
30. Fisheries and Oceans — The Effects of Salmon Farming in British Columbia on the Management of 


Wild Salmon Stocks 


St. Fisheries and Oceans — Fleet Management 

Be. National Defence — Defence Support Productivity: A Progress Report 
a, Follow-up of Recommendations in Previous Reports 

34. Other Audit Observations 

So) Sustainable Development Strategy for the Office of the Auditor General 


My Office issued the first volume of the 2000 Report in April, containing 9 chapters, and the second volume in 
October 2000, containing 8 chapters. 
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In addition, this year my Office has provided: 


- an auditor’s report and observations on the Financial Statements of the Government of Canada; 


- an auditor’s report and observations on the Statement of Transactions of the Debt Servicing and 
Reduction Account; and 


* some 90 auditor’s reports and observations on federal Crown corporations, federal departmental 
corporations and other entities; territorial governments, corporations and other entities; and 


international organizations. 


In 1999-2000, my Office completed special examinations of Standards Council of Canada, National Museum of 
Science and Technology, Canada Development Investment Corporation, Export Development Corporation, Marine 
Atlantic Inc., Old Port of Montreal Corporation Inc., Canada Deposit Insurance Corporation, Enterprise Cape 
Breton Corporation, National Gallery of Canada, Freshwater Fish Marketing Corporation, Queens Quay West 
Land Corporation and Royal Canadian Mint. Further, in the subsequent period from April to October 2000, 

my Office also completed special examinations of Laurentian Pilotage Authority, Canadian Diary Commission, 
Canadian Broadcasting Corporation and Canada Lands Company Limited. 


Under section 11 of the Auditor General Act, | may undertake, from time to time, assignments at the request of the 
Governor in Council. In 2000, we received the following requests under this section: 


* Inquire into the Yukon Workers’ Compensation Health and Safety Board. 


* Inquire into and report on the accounts and financial transactions of UNESCO, including any 
observations with respect to the efficiency of the financial procedures, the accounting system, the 
internal financial controls and, in general, the administration and management of the organization, 
for three consecutive fiscal years commencing in the year 2000-2001. 


* Audit, each year, the annual financial statements of the Canadian Nuclear Safety Commission. 


* Inquire and report on the reconciliation referred to in article 23 of the Sales Tax Collection 
Agreement between the Government of Canada and The Kamloops Indian Band. 
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Main Points 


Each year in this chapter I highlight issues I consider especially significant. In this, my tenth and 
last such chapter, I concentrate on the question of how we might improve the management of government 
spending. I discuss four basic principles that should be informing spending decisions and practices in a 
parliamentary system of government like our own: 


¢ All government spending should have Parliament’s sanction. Parliamentary control of 
finance has been described as the bedrock of responsible government. It is based on two 
central precepts of parliamentary democracy — namely, that the government should not raise 
money without Parliament’s approval, and it should not spend money except for purposes 
authorized by Parliament. Compliance with parliamentary authority is a minimum 
requirement that must be met by any disbursement of public funds. 


¢ Government spending should be managed with probity and efficiency. This principle 
recognizes the fact that public money is money held in trust for the benefit of all Canadians. 
As a consequence, the government has an obligation to ensure that the money is managed 
prudently in support of the general public interest. It also means that the government must 
seek to obtain maximum value for the dollars it spends. 


¢ The value of government spending should be measured by what it achieves. Historically 
the tendency has been to justify government programs more by the resources they used than 
by the results they produced. But governments don’t create programs in order to spend 


money; they establish programs to deliver services that improve the lives of Canadians. That 
is why they need to manage for results. 


Government spending programs should remain current. The resources available to 
government change over time. So do public needs and preferences. To ensure that public 
spending is allocated to the uses that are most valued, the government needs to evaluate its 
programs regularly and assess whether they are still relevant, effective and affordable. 


While these principles may appear self-evident, they are nevertheless too frequently breached. I 
therefore also discuss how a strong framework of values and ethics and appropriately structured 
incentives can help to ensure that these principles of public spending are observed. 
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In a supplement to the chapter, I address other matters of ongoing interest: 
* reforms in human resource management in the public service; 

* financial management capabilities in government departments; 

* progress in performance reporting by government departments; 

* weaknesses in the education system for Indian children on reserves; 

* challenges facing federal health and safety regulatory programs; 

¢ shortcomings in the corporate governance of Crown corporations; and 


* opportunities for cost savings identified by our audits. 
Background 


Despite the retrenchment and cutbacks of the past decade, spending by the federal government 
remains at historically high levels — roughly $175 billion a year. How well this money is spent is clearly 
of great interest to the taxpayers, who bear the cost, and to the public at large, on whose behalf the 
spending takes place. 


In my 10 years as Auditor General, I have seen significant improvements in the management of 
government spending, often made under the pressure of fiscal constraints. But progress has been slow, 
and there have been relapses as well. Year-end surges in spending, expenditure commitments without 
adequate planning or control, insufficient effort to recover moneys owed the Crown — these and other 
indications of a failure to make the best use of resources are still encountered with disquieting frequency 
in the federal public sector. 


Why do these problems seem so intractable? Why do they persist year after year, despite express 
commitments to set them right? What can be done to close the gap between promise and performance in 
the way government handles taxpayer money? These are the questions that motivate my discussion in 
this chapter. 
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We are committed to making a difference for the Canadian people by 


promoting, in all our work for Parliament, answerable, honest and 


Vision 


productive government that reflects a commitment 


to sustainable development. 


Mission 


The Office of the Auditor General of Canada conducts independent audits 
and examinations that provide objective information, advice and assurance 
to Parliament. We promote accountability and best practices in 


government operations. 


Elaboration of Mission 


In achieving our mission, 


we want to make a difference by promoting: 


® a fair and frank accounting of government's stewardship of financial 


and other resources; 


¢ 


efficiency and productivity in the public service; 


® 


cost effectiveness of government activities; and 


® collection of revenues owed to the Crown. 


Other effects we want to produce 


through our work are: 
® objective assurance on matters found to be satisfactory and 
unsatisfactory; 
® compliance with authority; and 


® deterrence of fraud and dishonesty. 
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Improving Performance in Government Spending 


ach year in this chapter I highlight 
(Ee issues I consider especially significant. 

This is my tenth and last such chapter: 
my 10-year mandate as Auditor General comes to 
a close at the end of March 2001. In February, I 
plan to present a “capstone” report to Parliament 
to coincide with the end of my mandate. That 
report will provide my perspective on some key 


management problems facing the federal 
government, the role of legislative audit and its 
evolution over the past 10 years, and the efforts my Office has made to 
identify problems and seek reform in central agencies and in program 
delivery. 


: In this chapter I want to concentrate on a theme that is 
Despite honest efforts and repeated central to our Office and dear to my heart: how to 


commitments to improvement, improve the way spending is managed in the public 


expenditure management in the sector. Despite honest efforts and repeated commitments 
to improvement, expenditure management in the federal 


federal public sector is still not as public sector is still not as good as it should be and can 
good as it should be and can become. __ become. To this end, I set out some basic principles that 
should be informing spending decisions and practices in a 
parliamentary system of government like our own, and I 
draw out some of their implications. I also discuss how a strong framework 
of values and ethics and appropriately structured incentives can help to 
ensure that these principles of public spending are observed. 


The stewardship of public funds deserves increased attention 


Despite the retrenchment and cutbacks of the past decade, spending by the 
federal government remains at historically high levels — roughly 

$175 billion a year. How well this money is spent is clearly of great interest 
to the taxpayers, who bear the cost, and to the public at large, on whose 
behalf the spending takes place. 


Earlier this year, the disclosure of serious shortcomings in the management 
of grants and contributions at Human Resources Development Canada 
(HRDC) created a major stir. Public reaction was strong because the amounts 
involved were large and also, I believe, because years of rising taxes and 
shrinking or stagnating household budgets have made the public even more 
sensitive than usual to lax spending practices in government. The issue 
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received wide publicity in the news media and was intensely debated in the 
House of Commons. But shortcomings of the sort revealed at HRDC — 
vague and inconsistently applied eligibility criteria, breaches of authority, 
absence of an appropriate control and accountability framework — are by no 
means exclusive to one program or one department. We observed 
shortcomings of a similar nature (though of a lesser magnitude) in our 1996 
audit of the Canada Infrastructure Works program, our 1997 audit of The 
Atlantic Groundfish Strategy and our 1998 audit of grant and contribution 
programs in Industry Canada and the Department of Canadian Heritage. I can 
cite other examples over the years, but these will suffice to make my point 
that spending in government is not always conducted with the care and 
prudence that taxpayers have a right to expect. 


Canadians are justifiably upset by such disclosures. 


Spending in government is not They may agree with Justice Oliver Wendell Holmes 


always conducted with the care and 


that taxes are the price we pay for civilization, and not 
begrudge the high taxes they pay to finance necessary 


prudence that taxpayers have a right public services. But they get upset and angry when 


to expect. 


they see their tax dollars squandered or used with less 

care and prudence than they must use to make ends 
meet in their household budgets. Frankly, I share their frustration. It is 
discouraging to witness new incidents of waste and mismanagement crop up 
hydra-like after older ones have been discovered and dispatched. 


In my 10 years as Auditor General, I have seen significant improvements in 
the management of government spending, often made under the pressure of 
fiscal constraints. But progress has been slow, and there have been relapses 
as well. Year-end surges in spending, expenditure commitments without 
adequate planning or control, insufficient effort to recover moneys owed to 
the Crown — these and other indications of a failure to make the best use of 
resources are still encountered with disquieting frequency in the federal 
public sector. 


Why do these problems seem so intractable? Why do they persist year after 
year, despite express commitments to set them right? What can be done to 
close the gap between promise and performance in the way government 
handles taxpayer money? 


If these questions were easy to answer, we would not need to revisit them so 
often. But after 10 years of looking close up at government spending, I 
believe we can do better and should resolve to do so. The core of this chapter 
discusses how this might be done. The section that follows sets out a number 
of principles that ought to guide spending decisions in government. 
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Subsequent sections of the chapter discuss means of reinforcing the 
principles through appropriate structures of incentives and ethics. As in 
previous years, issues that are not directly related to the theme of the chapter 
(in this case, ensuring prudent spending in the public sector) but that I 
consider important are reported in a supplement to the chapter. 


Four Basic Principles of Public Spending 


Public administration at the federal level in Canada has undergone huge 
changes over the past decade. As in many other countries in the developed 
world, there has been a marked shift away from the traditional bureaucratic 
emphasis on detailed control from the cenire and preoccupation with process 
toward more autonomy for departments and a focus on results. The shift to 
results-oriented management has been driven by a desire to harness the 
enterprise and judgment of public sector employees and 


The downside risks of empowerment empower them to ALI ghia ECS At the same time, 
b nimized if k however, the weakening of traditional controls can increase 
can pe minimized if some Key 


principles are applied to help guide 
action and set limits on the exercise 
of discretion. command-and-control approaches do not work — if they 


the risk of error, misjudgment or abuse. 


Reverting to the old rule-bound systems is not an answer to 
this dilemma. In today’s information age, top-down, 


ever did. The downside risks of empowerment can be 
minimized if some key principles are applied to help guide action and set 
limits on the exercise of discretion in the new, decentralized environment of 
the public sector. 


My 1992 Report stressed the need for appropriate safeguards to accompany 
the devolution of authority in the public sector. Drawing on parallels with 
private sector firms, the Report suggested that a set of fundamental rules be 
instituted to frame the boundaries of decision making by public servants. A 
clear set of ground rules within which discretion can be exercised, the Report 
contended, can help to minimize unwanted risks while encouraging 
innovation by providing more certainty about what employees may do. 


The rules my 1992 Report suggested are shown in Exhibit 1. The list is not 
exhaustive; other examples can be added. To work, the rules would have to 
be well understood. They should therefore be codified and communicated 
throughout the public service. I still believe that this would help to minimize 
unwanted risks in government activities. 


These rules, and others like them, can be captured in a small number of 
principles rooted in our parliamentary system of government and in the 
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government’s fiduciary obligation to Canadians for the management of 
public funds. I discuss four such principles here: 


Exhibit 1 — Some Fundamental Rules for the Public Service 


Do nothing illegal. Uphold the spirit and letter of laws and 
regulations. 


Act within your delegated authority. 

Act in a politically neutral manner. 

Act impartially, honestly and fairly. 

Don’t waste public money. 

Safeguard assets. 

Respect the core values of your organization. 
Make service standards visible. 


Realize the productive potential of all resources entrusted, 
especially people. 


Don’t make decisions that commit significant public resources 
without proper authority. 


Document risk acceptance. 


Account for the discharge of responsibilities and achievement 
of agreed performance. 


Get authority for waiving a fundamental rule. 


Source: Report of the Auditor General of Canada, Chapter 4, 1992. 


e All government spending should have 
Parliament’s sanction. 


e Government spending should be 
managed with probity and efficiency. 


e The value of government spending 
should be measured by what it achieves. 


e Government spending programs should 
remain current. 


While these principles may seem 
self-evident, they are apparently not clear 
enough to have prevented breaches of 
each one of them, as I note below. I 
believe that further emphasis on guiding 
principles and fundamental rules would 
reduce the incidence of such breaches in 
the future. 


I also discuss procedures and practices 
that flow from these principles. The 


procedures and practices may vary over time; the principles themselves 


remain constant. They can give direction to specific processes and inform 


appropriate conduct. 


All Government Spending Should Have Parliament’s 


Sanction 


Parliamentary control of finance has been described as the bedrock of 
responsible government. It is based on two central precepts of parliamentary 
democracy — namely, that the government should not raise money without 
Parliament’s approval, and it should not spend money except for purposes 
authorized by Parliament. These precepts, the achievement of centuries of 
struggle between the Crown and the Commons in England, were already well 
established at the time of our Confederation in 1867. When the original 
provinces got together to establish a form of government “similar in 
Principle to that of the United Kingdom,” they incorporated these precepts in 
the Constitution Act, 1867. The principle that Parliament is the custodian of 
the public purse has thus been part of Canada’s constitutional landscape since 


the country’s inception. 
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Over time, a number of specific practices were developed to put this 
principle into effect. Since 1951, those practices have been brought together 
under a single act of Parliament, the Financial 

Dey. : : Administration Act (FAA). The Act makes the Treasury 
The principle that Parliament is the Board responsible for overall control of public spending 


custodian of the public purse has and each minister responsible for the legality and probity 


been part of Canada’s constitutional of spending in his or her department. It also sets precise 
land : th try’ conditions that govern payments out of the Consolidated 
Sec eM ie eye Revenue Fund — the account into which, by law, all 


inception. government revenues are credited. Specifically, the 
payments must: 


e be authorized by a warrant signed by the Governor General (FAA, 
section 28); 


e conform to a plan approved by the Treasury Board (section 31); 


e not exceed funds appropriated for the purpose by Parliament 
(section 32); and 


e be requisitioned by a minister or authorized officer in a form prescribed 
by the Treasury Board (section 33). 


Where payments are for services or goods received, the responsible 
departmental officer must certify before payment is made that the 
performance conditions were met. For grants and contributions, the officer 
must certify that the recipient is entitled to the payment (section 34). 


These provisions of the FAA establish minimum requirements for the 
disbursement of public funds — there are no short-cuts around them. Yet 
breaches of these provisions, while infrequent, still occur. In our audit of 
HRDC earlier this year, we found instances of payments made without 
required approvals or before project commitments had been met. The audit 
found that payments had been approved by staff who lacked the delegated 
authority to do so and who often did not review whether claims for payment 
were supported by the requisite information. The point of the FAA provisions 
is to ensure that government spending is in compliance with parliamentary 
authority and consistent with enunciated government policies. Failure to 
adhere to them normally signals further weaknesses in the way a program is 
managed and run. 


For related Auditor General work, see: 2000 Chapter 11 Human Resources 
Development Canada —Grants and Contributions; 2000 Chapter 17 Other Audit 
Observations — Parc Downsview Park Incorporated and Public Works and 
Government Services Canada; 1998 Chapter 27 Grants and Contributions: Selected 
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Public money is money held in trust 
for the benefit of all Canadians. maximum value for the dollars it spends. Common 


Programs in Industry Canada and Department of Canadian Heritage; 1997 
Chapter 13 Health Canada —First Nations Health; 1996 Chapter 26 Canada 
Infrastructure Works Program —Lessons Learned; I 994 Chapter 12 Aspects of 
Federal Real Property Management; 1993 Chapter 25 Parliamentary Control over 
the Raising of Revenues by Fees. 


Government Spending Should Be Managed With Probity 
and Efficiency 


This principle recognizes the fact that public money is money held in trust 
for the benefit of all Canadians. As a consequence, the government has an 
obligation to ensure that the money is managed 
prudently in support of the general public interest. It 
also means that the government must seek to obtain 


sense and good management practices suggest a 
number of conditions that need to be in place if these fiduciary obligations of 
the government are to be met. They include appropriate program design, 
adequate capacity to run each program, and effective systems of internal 
control. 


Appropriate design. Legislation normally provides only general direction 
and outlines of spending plans. Departmental managers are then responsible 
for designing specific programs to achieve the legislation’s intent. They need 
to define clearly the desired outcomes and set priorities; choose appropriate 
mechanisms and instruments to deliver the program; and identify potential 
risks and means to manage them. 


Haste makes waste. The Northern Cod Adjustment and Recovery Program 
offers a good example. Established in 1992 to provide relief to fishers and 
plant workers affected by the moratorium on cod fishing, the program was 
rushed into operation before proper eligibility criteria and appropriate 
financial controls had been established. As a result, payments were made to 
many people who were not affected by the moratorium; and contrary to the 
program’s intent, people who had left the fishery were drawn back to it. 
Spending the time and the resources it takes to establish a workable, effective 
plan for running a program is an investment worth making. 


Adequate capacity to run the program. Does the program have enough 
resources to achieve the desired results efficiently? Are they allocated 
optimally across the tasks involved in running the program? Having adequate 
capacity in place is a manifestly basic ingredient of program success, but one 
not always present in government departments. As we reported in October, 
inadequate capacities for financial management and control at HRDC in the 
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wake of the cutbacks and reorganizations of the mid-1990s were factors that 
contributed to the mismanagement of several programs in that department. 
Similarly, our 1996 audit of the Canada Pension Plan 
(CPP) disability program concluded that the sharp 
Establishing huge spending increases in CPP disability payments during the previous 


programs and then, in effect, putting decade had occurred in the absence of sufficient capacity 
nay to analyze trends in benefit payments, manage eligibility 
them on auto-pilot IS clearly not assessments, and review files systematically to identify 
consistent with the principle of ineligible beneficiaries. Establishing huge spending 
seeking to maximize value for the programs and then in effect putting them on auto-pilot is 
: clearly not consistent with the principle of seeking to 
taxpayer's dollar. maximize value for the taxpayer’s dollar. A program 
worth doing is worth doing well. Inadequate resources or 


resources poorly deployed can undermine the effectiveness of the program 
and lead to waste, loss of trust and erosion of support for the program. 


Effective internal control systems. Internal control refers to the methods 
and procedures used by management to ensure that decisions are carried out 
properly, risks are managed, transactions recorded, performance monitored 
and results reported: in short, everything it takes to ensure that the program is 
running as it should. 


An integral part of an effective internal control system is a strong audit 
function, one that tests the credibility of performance measurements and 
provides assurance that the management framework for achieving program 
objectives is operating as intended. In government-wide audits completed in 
1993 and 1996, we found weaknesses in the internal audit functions of many 
departments. Since then, disproportionate cutbacks in the government’s 
internal audit, evaluation and review functions have eroded them further. In 
HRDC, for example, the internal audit staff was reduced from 54 in 1994-95 
to 33 in 1999-2000. As Treasury Board standards recognize, internal audit is 
a fundamental tool of effective management. Treating it as a frill or an 
afterthought can put an organization at risk. 


For related Auditor General work, see: 2000 Chapter 3 Citizenship and Immigration 
Canada —The Economic Component of the Canadian Immigration Program; 2000 
Chapter 13 Assessment of Financial Management Capabilities in Departments; 2000 
Chapter 14 Canadian International Development Agency —Managing Contracts 
and Contribution Agreements; 2000 Chapter 18 Governance of Crown Corporations; 
1996 Chapter 4 Internal Audit in Departments and Agencies; 1996 Chapter 17 
Human Resources Development Canada —Canada Pension Plan: Disability; 1993 
Chapter 7 Internal Audit; 1993 Chapter 15 Department of Fisheries and Oceans — 
Northern Cod Adjustment and Recovery Program. 
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Historically the tenden 


justify government programs more by 


The Value of Government Spending Should Be Measured 
By What It Achieves 


Defining clearly the results of a program is a difficult task — especially in 
the public sector, where the policy aims are often intrinsically broad — but a 
task that is necessary. Historically the tendency has 
cy has beento __ been to justify government programs more by the 
resources they used than by the results they produced. 
But governments don’t create programs in order to 


the resources they used than by the spend money; they establish programs to deliver 


results they produced. 


services that improve the lives of Canadians. What is a 

program expected to achieve? How will we know if it 
does? These questions need to be answered as clearly as the circumstances of 
each program will permit. That is why government needs to manage for 
results. 


As noted in Chapter 20 of this Report, managing for results in an 
organization requires the following elements: 


e Fostering an organizational climate that encourages managing for results, 
including leadership from senior management and appropriate incentives. 


e Agreeing on a strategic framework for results, making clear what the 
organization intends to accomplish and how. 


e Agreeing on the expected results and strategies of programs, establishing 
clear objectives and performance indicators up front. 


e Measuring performance results and using that information to improve the 
design and operation of the program. 


e Reporting performance effectively, that is, in a way that allows 
Parliament and stakeholders to see how well the program is doing. 


Establishing clear objectives and performance indicators up front and regular 
monitoring and reporting of results are useful on several counts. They 
encourage program improvements by helping management know how well a 
program works. They enhance Parliament’s ability to hold the government to 
account for the use of public funds. And they nourish public trust in 
government, by enabling the public to know what they get for their tax 
dollars and whether it represents the best value for the money. 


The government does recognize the importance of managing for results. In a 
November 1995 paper entitled Strengthening Government Review, it 
announced an explicit commitment to establish “a results-based management 


Report of the Auditor General of Canada — December 2000 


Matters of Special Importance — 2000 


culture,” embracing the expectation that “programs and 
Establishing clear objectives and services will identify intended results, measure progress 
performance indicators up front and and then report on achievements.” As we report in 
oe ; Chapter 20, the government has introduced several 
regular monitoring and reporting of important initiatives in support of that commitment, and a 


results are useful on several counts. focus on results is now common in government 
departments. However, it is proving very difficult to 
complete certain critical steps, such as measuring results and using that 
information to improve performance. 


For related Auditor General work, see: 2000 Chapter 20 Managing Departments for 
Results and Managing Horizontal Issues for Results; 2000 Chapter 29 Treasury 
Board Secretariat —the Government's Annual Contracting Activity Report — 1998; 
1999 Chapter 19 Industry Portfolio —Investing in Innovation; 1997 Chapter 5 
Reporting Performance in the Expenditure Management System; 1997 Chapter 11 
Moving Toward Managing for Results; 1997 Chapter 22 Crown Corporations: 
Making Performance Measurement Work. 


Government Spending Programs Should Remain Current 


The resources available to government change over time. So do public needs 
and preferences. To ensure that public spending is allocated to the uses that 
are most valued, government needs to evaluate its programs regularly and 
assess whether they are still relevant, effective and efficient. Questions that 
evaluations should address include the following: 


e Does the program still serve a clearly defined public purpose that matters 
to Canadians? 


e Would we establish the program today if it did not already exist? 
e Is it desirable to maintain it at the current level? 


e Can it be delivered more effectively or efficiently? Have there been 
changes (in the service environment infrastructure, technology, etc.) 
since the program’s inception that would now permit an alternative 
means of achieving its objectives with greater economy, efficiency, or 
effectiveness? 


These questions are similar to those asked during the government’s Program 
Review that preceded the deficit-cutting measures announced in the February 
1995 Budget. The government claims that Program Review was instrumental 
in establishing the priorities that should be supported and in identifying 
where spending could be cut with the least pain to the system and fewest 
adverse consequences for Canadians. These are, of course, precisely the 
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purposes that program evaluation can serve. But evaluations should not be 
sporadic, or byproducts of financial crisis. Systematic, ongoing evaluation of 
programs is an essential part of good public management. 


High deficits and the need to rein them in provided strong incentive during 
the 1990s to review government programs closely for savings opportunities 
and to evaluate new spending proposals carefully. 


Systematic ongoing evaluation of Indeed, under the government’s Expenditure 
b] 


programs is an essential part of good 


public management. 


Management System published in early 1995, all new 
spending had to be financed through reallocations 
from existing programs. In the current post-deficit 
environment, the demand for fiscal discipline may 
appear less pressing. The need for ongoing review of government programs 
remains, however, for we still need to ensure that they continue to serve a 
useful public purpose and that they remain efficient, effective and affordable. 


For related Auditor General work, see: 1996 Chapter 3 Evaluation in the Federal 
Government; 1993 Chapter 8 Program Evaluation in the Federal Government — The 
Case for Program Evaluation; 1993 Chapter 9 Program Evaluation in Departments 
—The Operation of Program Evaluation Units; 1993 Chapter 10 The Program 
Evaluation System — Making It Work. 


Recent government pronouncements on spending practices are 
commendable, but follow-through is key 


I was encouraged to see the government’s clear commitment to “responsible 
spending” in the Treasury Board publication, Results for Canadians: A 
Management Framework for the Government of Canada, released earlier this 
year Specific parts of that commitment coincide with several of the 
prerequisites for sound management of public spending I have been 
discussing here, including a focus on results, a proper environment of 
control, and continuous examination of expenditures to ensure that value is 
obtained for the taxpayer’s dollar. 


I was also pleased at the revised policy for transfer payments announced last 
May by the Treasury Board President. Key components of that new policy — 
clear eligibility criteria, due diligence in approving entitlement and making 
payments, a results-based accountability framework, enhanced reporting to 
Parliament, ongoing monitoring and evaluation — are consistent with 
recommendations we have made in past reports on audits of grant and 
contribution programs. Appropriately implemented, the revised policy should 
result in much better management of government transfer payments. 
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Effective implementation is key of course. In the rest of this chapter, | 
discuss how a sound framework of values and ethics and the right incentives 
can help convert principle into practice. 


A Strong Foundation of Values and Ethics Can 
Support Prudent Spending 


Government is increasingly relying on values and ethics, rather than on 
prescriptive controls, to maintain probity and prudence in the public sector. A 
strong foundation of values and ethics helps to ensure that sound judgments 
are made in deciding how to spend money 
Exhibit 2— Components of an Ethical Framework and implement programs. There is such a 
foundation in the federal public service at 


e Statement of principles. 


e Leadership by ministers and senior officials. present, but it should not be taken for 


e Empowerment of public servants to carry out their duties in the granted. 
public interest. 
e Increased transparency in decision making. As we note in Chapter 12 of this year’s 
e Provision of ethics-related training. Report, major changes in government, such 
e Effective and trustworthy recourses for voicing ethical as the relaxation of central controls and the 
ph diffusion of decision-making authority, could 
Source: Reports of the Auditor General of Canada, 1995 and 2000. undermine the existing strong foundation of 


values and ethics. We have proposed a 
framework to help maintain that foundation. This framework focusses on 
leadership, transparency in decision-making and the provision of guidance 
and training (see Exhibit 2). 


The government has taken a number of important steps in recent years to 
maintain a strong foundation of values and ethics in the federal public sector. 
The 1996 report of the Task Force on Public Service Values and Ethics (the 
Tait report) helped forge a consensus on the current state of values and ethics 
in the federal public service and on areas for action. As well, the government 
is trying to reinvigorate a discussion of values and ethics in the public 
service. A number of departments are in the process of implementing 
sophisticated programs of values and ethics. For the most part, however, 
work to implement needed reforms in this area is only at a preliminary stage. 


In its efforts to sustain sound values and ethics in the public sector, the 
government faces several major challenges. A set of principles to guide the 
public sector has to be developed. This cannot be imposed. It needs to be 
developed with the full participation of members of the public service and 
those who do business with the government. If the principles are to have 
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In its efforts to sustain sound values 
and ethics in the public sector, the 


credibility, leadership by example throughout the public sector will be 


critical. 


The government is telling public servants that ethics 
involve “a commitment to do the right thing.” It is 
also advising them that in deciding how to act, they 


government faces several major should consider their obligation and authority to act, 


challenges. 


choose the best option, and accept responsibility for 

the actions taken. To help public servants act on this 
advice, the government needs to clarify the responsibilities of ministers and 
officials so there is a better understanding of who is accountable for what. 


There is also a need to provide public servants with effective means to 
intervene in good faith and voice ethical concerns. Public servants should not 
be put in the position of believing that their only recourse is to express their 
concerns publicly. It should be possible to address matters of ethical concern 
effectively within an organization. To make it possible, the recourse 
mechanisms must be clear and trustworthy. 


Finally, organizations and individuals doing business with the government or 
receiving government funds must play a supportive role. They need to 
understand that they are expected to respect the values and ethics that 
underpin sound public administration. The government will need to provide 
appropriate guidance on this. 


A strong foundation of sound values and ethics is an essential part of good 
governance. It deserves more vigorous and systematic support than it has 
received to date. The government need to show greater determination on this 
front, develop a plan to address the issues involved, and act decisively to 
implement it. 


For related Auditor General work, see: 2000 Chapter 12 Values and Ethics in the 
Federal Public Sector; 1999 Chapter 26 National Defence —The Proper Conduct of 
Public Business; 1998 Chapter 15 Promoting Integrity in Revenue Canada; 

1995 Chapter I Ethics and Fraud Awareness in Government. 
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Incentives Should Be Harnessed to Pull in the 
Right Direction 


A strong commitment to the values of professionalism, integrity and service 
reduces the risk that office holders will behave improperly or abuse the 
power of their office. But virtue alone is not enough to produce good results. 
As the famous British economist Alfred Marshall once observed, “Progress 
chiefly depends on the extent to which the strongest and not merely the 
highest forces of human nature can be utilized for the increase in social 
good.” An altruist will give more to charity than a miser, but each is likely to 
give less when the cost of giving rises (through a reduction in the tax subsidy 
afforded charitable donations, for instance). Incentives matter. How to make 
them work in the administration of the public sector is a question of the first 
importance. Regrettably, it has not always received the attention it deserves. 


Both carrot and stick can be used for the purpose. Deliberate abuse of power 
must invite appropriately corrective consequences. Good performance — 
performance above and beyond the call of duty — should 


The question of the right incentives be rewarded accordingly. Opportunities to tie rewards to 
eee performance may not be as prevalent in the public sector 
extends beyond individual rewards as in the private sector, but they do exist. Bonuses, 
and sanctions to broader rules in promotions and reassignments are some available means. 
government. Programs of public recognition for a job well done or 
~~ srecognition from colleagues can also motivate a greater 


effort to improve performance and are often easier to implement than 
monetary incentives. 


The question of the right incentives extends beyond individual rewards and 
sanctions to broader rules in government. To illustrate, until 1993 all savings 
achieved by departments — any unspent balances by year-end — would be 
returned to the government’s general coffers. Where was the incentive to 
save in such a system? In 1993, the supply process was amended to allow 
departments to carry forward a portion of their unspent funds to the 
following fiscal year. As a result of that change, year-end spending sprees by 
departments are now less common than they used to be. 


Unfortunately, they have recently been replaced by a similar practice at a 
more aggregate level. With the elimination of the deficit three years ago, the 
government has adopted the practice of announcing significant new spending 
toward year-end so the estimated surplus is reduced to a level no higher than 
the $3 billion “contingency reserve” included in the Budget forecast. Largely 
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There is a tendency in government at 


as a result of this practice, cumulative spending in excess of Budget 
projections has amounted to $10.5 billion over the past three years. 


Poorly structured incentives also appear to have played a role in the 
mismanagement of job creation programs at HRDC. According to testimony 
before the House of Commons Standing Committee on Human Resources 
Development and the Status of Persons with Disabilities, HRDC front-line 
employees were under pressure to approve projects quickly and spend the 
available funds. Employees who exercised prudence might have been 
assessed poorly for failing to disburse enough funds. 


Mistakes can occur in any organization. When they 


times to move cautiously in ferreting occur in the private sector, there are strong 
out error and to respond slowly when incentives to correct them quickly and move on: 


it is found. 


Enhanced transparency and 


financial markets and the bottom line are powerful 
enforcers. 


In the public sector, rectifying mistakes often appears less pressing. To set a 
failure right, it is necessary to own up to the failure first. But where are the 
benefits to ministers and senior management of acknowledging error? The 
fallout can often be severe. Cover the mistake up, on the other hand, and life 
chugs along normally: none is the worse off but taxpayers. There is therefore 
a tendency in government at times to move cautiously in ferreting out error 
and to respond slowly when it is found. 


A number of lessons emerge from this experience: 


e It takes courage for public officials to bring administrative abuses and 
mistakes to light. When they do, they deserve protection against possible 

retribution from their bureaucratic or political 

masters for the discomfort they may have caused. 


accountability lead to stronger As the Task Force on Public Service Values and 


institutions and more effective 


government. 


Ethics concluded, appropriate mechanisms are 
needed to permit public servants to express dissent 
and raise ethical concerns without suffering for their 
efforts. 


e Accountability requires that people accept responsibility for their 


mistakes — that goes without saying. Excessive emphasis on laying 
blame, however, can be counterproductive. If we wish to empower 
employees and encourage them to innovate, then we must be prepared to 
accept the risk that, at times, mistakes and wrong decisions will be made. 


When that happens, we should focus on learning from the experience 
rather than assigning blame. 
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e Parliament and the public should encourage more openness in 
government. Sunlight, it has been said, is the best antiseptic. Clear, 
timely information on money spent, where it went and what it achieved 
makes it easier for those outside the organization to monitor the 
disbursement of funds for consistency with policy intentions, probity and 
prudence. The knowledge that their operations are visible encourages 
public servants and their political masters to exhibit behaviour that can 
withstand scrutiny. Overall, enhanced transparency and 
accountability lead to stronger institutions and more effective 
government. 


For related Auditor General work, see: 1999 Chapter 11 Agriculture Portfolio — 
User Charges; 1994 Chapter 32 Department of Finance and Revenue Canada — 
Income Tax Incentives for Research and Development; 1992 Chapter 5 The Learning 
Organization. 


Conclusion 


The past decade has seen a strong movement toward a less bureaucratic, 
more flexible approach to management in the federal public sector. The 
relaxation of central controls, increased managerial discretion, a focus on 
results over process — these are all elements in this trend. This is a trend that 
I very much favour and have strongly supported, but it has a downside. As 
recent disclosures of some lax spending practices in government suggest, 
when controls are reduced and more discretion is allowed, there is some risk 
that discretion will be misused and things may go wrong. 


To minimize that risk, devolution of authority must be matched with some 
basic principles to guide spending decisions and to place appropriate bounds 
on the use of discretion. In broad terms, these principles require acting within 
authority and acting with prudence while maintaining a focus on results that 
matter to Canadians. Reinforced with appropriate incentives, values and 
ethics, they can help produce government that is both frugal and effective. 


The Canadian economy has been on a strong upward trajectory over the past 
few years, and government coffers are flush with cash. Good times are not 
normally conducive to intensive efforts to achieve economy and efficiency. 
All the more reason, therefore, for parliamentarians and the public to remain 
vigilant, and to insist that government spending take place within a 
framework that respects the supremacy of Parliament and fosters value for 
the taxpayer’s dollar. 
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Supplement: Other Matters 


Reforms in human resource management 


have not met expectations 


Public management in Canada has changed 
significantly in the last decade, becoming less reliant 
on detailed central controls and more focussed on 
results. As part of this change, departments and their 
employees have received broader authority and 
discretion to find innovative ways of meeting 
objectives. These changes have highlighted the 
importance of improving the way government 
employees are managed in order to attain desirable 
results. As I have stressed on previous occasions, 
success on this front is key to successful public 
sector reform generally, for it is people who must 
implement the reforms, deliver the services and 
provide feedback for improved performance. 


Despite major initiatives during the 1990s, human 
resource management reforms have not met 
expectations. Further, the challenges have taken on 
increased urgency. One reason is that measures to 
cope with difficult fiscal circumstances exacerbated 
poor labour-management relations and led many 
public servants to question the wisdom of a career in 
the public service. A second is that the public service 
faces new or heightened concerns. These include an 
impending “retirement bulge”, which raises the 
spectre of a future leadership crisis, and increasing 
competition from other employers for “knowledge 
workers.” 


There is a need for a real breakthrough in human 
resource management if the public service is to meet 
the challenges before it. Key to achieving this 
breakthrough is the resolution of three long-standing 
issues. The administrative framework governing the 
public service is cumbersome, costly and ill-suited to 
the demands of modern management. Responsibility 
is fragmented to an extent that clear leadership is 
lacking and accountability for reforms is unclear. 
And the increasing autonomy of departments 
demands major changes in the roles played by 
departmental managers. Deputy ministers, 
traditionally seen as primarily responsible for 
administering a centrally prescribed framework, 
must instead assume pivotal responsibility for the 
way their people are managed and for maintaining a 
healthy work environment in their departments. 


For related Auditor General work see: 2000 Chapter 
9 Streamlining the Human Resource Management 
Regime: A Study of Changing Roles and 
Responsibilities; 2000 Chapter 21 Post-Secondary 
Recruitment Program of the Federal Public Service; 
2000 Chapter 22 Development of the Universal 
Classification Standard: A Follow-up. 


Financial management capabilities in 


departments fall short of what is needed 


Encouraging better financial management in government 
has been a long-standing strategic priority of my Office. 
As part of this effort, in April 1999 we published the 
Financial Management Capability Model, a framework 
describing the essential elements of financial 
management in government departments and agencies. 
The Model sets out the Office’s expectations for 
financial management and is the basis on which future 
audits in this area will be conducted. 


In Chapter 13 of this year’s Report, we presented the 
results of an audit of the financial management 
capabilities in five government departments. The audit 
found that none of these departments meets the 
requirements of important current government 
initiatives, such as the Modernization of Comptrollership 
and the Financial Information Strategy. These initiatives 
aim to modernize the way government manages 
resources, by strengthening internal controls and 
providing better information to support planning, 
decision making and reporting. 


One of the most common weaknesses we found was the 
limited ability to integrate financial and non-financial 
information. Without integrated information, 
departments cannot determine accurately how much it 
costs to produce a particular result or to deliver a service 
at a given level. They need such information to make 
sound decisions, manage risks well and account properly 
for the use of resources. 


It will be difficult to modernize government operations 
without strong financial management capabilities. The 
development of these capabilities throughout 
government will require the sustained support and 
concerted effort of the Treasury Board Secretariat, the 
Privy Council Office and deputy ministers. With 
important government initiatives hinging on capabilities 
that do not currently exist, there is no time to spare. 


For related Auditor General work, see: 2000 Chapter 13 
Assessment of Financial Management Capabilities in 
Departments; April 1999 Financial Management 
Capability Model; 1999 Chapter 21 Financial 
Information Strategy: Departmental Readiness; 1998 
Chapter 18 The Financial Information Strategy: A Key 
Ingredient in Getting Government Right. 
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Performance reporting has improved but 


progress has been slow 


In Chapter 19 of this Report, we examine the reporting 
of departmental performance to Parliament. Some 
improvement has been made since our 1997 audit of 
this matter, but I am disappointed at the slow pace of 
progress. While today’s performance reports are better 
than the information that was available to 
parliamentarians in the previous Part III Estimates, they 
are still too much a report of activities rather than an 
accounting of the results achieved compared with what 
was planned. Too often the planned results are vague 
about what the departments actually expect to achieve, 
and the reporting of performance does not link 
achievements to the longer-term benefits for Canadians. 


Members of Parliament and the public need good 
performance information to be able to hold the 
government to account. Parliamentarians need this 
information in their committee reviews of departmental 
programs and in their scrutiny of the budgetary process. 
Parliament may want to consider legislation to ensure 
that it is provided. But other changes are also needed in 
the reporting regime. Agreement on principles of 
reporting, more incentives for good reporting, and 
stronger oversight and challenge by the Treasury Board 
Secretariat are all required. Departments also need 
performance information to improve programs and 
services. 


I am pleased to note that a general focus on results 
seems to have taken hold in the culture of departmental 
management and performance reporting. But effective 
implementation requires that the next steps — 
measuring results, using that information to improve 
programs and credible reporting on results to 
Parliament — be taken with some urgency. 


For related Auditor General work, see: 2000 Chapter 
19 Reporting Performance to Parliament: Progress Too 
Slow; 2000 Chapter 20 Managing Departments for 
Results and Managing Horizontal Issues for Results; 
1997 Chapter 5 Reporting Performance in the 
Expenditure Management System. 
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Education system fails to meet the needs of 


Indian children on reserves 


In April of this year, we reported on our audit of 
elementary and secondary education for Indian children 
who live on reserves. Indian and Northern Affairs 
Canada (INAC) allocates about $1 billion each year for 
this purpose, in addition to funding school construction 
and maintenance. 


The audit found serious gaps in the academic 
achievement of First Nations students compared with 
other Canadians. The dropout rate of these students 
before grade nine is six times higher than that of the 
general Canadian population. The proportion of the 
on-reserve population with a high school education is 
significantly lower than in the general population. 
Moreover, it is questionable whether the education 
received by Indian students meets their cultural needs. 


This poor record reflects deficiencies in the education 
programs funded by INAC that have existed for a very 
long time. For one, the Department has not articulated 
its role in education. Clarification of the role is essential 
both for accountability purposes and for the Department 
to know whether its existing capacity and resources are 
sufficient to fulfil that role. 


The Department also needs to address several 
operational issues, including how education funds are 
administered. As a result of program devolution and 
delegation of authorities, the Department has come to 
rely on First Nations and the provinces for the delivery 
of education services. The Department needs to take the 
measures necessary to ensure that these services — for 
which it provides funding — are effective and efficient. 


Action to meet the education needs of Indian children is 
urgently needed. In the absence of satisfactory progress, 
there will be high financial cost in social programs, an 
increased waste of human capital, and degradation of 
the relationship between the government and First 
Nations peoples. 


For related Auditor General work, see: 2000 Chapter 4 
Indian and Northern Affairs Canada —Elementary and 
Secondary Education. 
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Health and safety regulatory programs face 


difficult challenges 


Health and safety regulatory programs deal with issues 
ranging from food inspection and nuclear power 
regulation to environmental pollution and air safety. 
Their basic purpose is to protect Canadians from risks 
to health and safety. This task is becoming 
increasingly complex, requiring a correspondingly 
sophisticated approach to accomplish it. 


Federal health and safety regulatory programs have 
experienced profound changes over the past decade. 
They were restructured to streamline delivery, and they 
shifted orientation away from direct regulation 
(designed and imposed by government) toward greater 
reliance on standards developed by third parties, such 
as voluntary industry codes. They were also subjected 
to funding reductions, which have been partly 
reinstated recently. 


Some of these changes have had an adverse impact on 
public confidence in the effectiveness and credibility 
of health and safety regulatory programs. Restoring 
public confidence is one of several serious challenges 
facing these programs today. Other challenges include 
a significant exposure to legal claims for regulatory 
negligence, arising from court decisions over the past 
decade; the need to co-ordinate operations across 
departments and jurisdictions, since many health and 
safety issues cut across departmental mandates and 
political borders; and the need for large-scale 
recruitment of expert staff, in light of the high 
proportion of employees who will be eligible for 
retirement in the near future. 


In Chapter 24 of this Report, we make a number of 
recommendations that can help government respond to 
these challenges. 


For related Auditor General work, see: 2000 Chapter 
24 Federal Health and Safety Regulatory Programs; 
2000 Chapter 25 Canadian Food Inspection Agency — 
Food Inspection Programs; 2000 Chapter 26 Health 
Canada — Regulatory Regime of Biologics; 2000 
Chapter 27 Canadian Nuclear Safety Commission — 
Power Reactor Regulation; 2000 Chapter 28 
Follow-up of Previous Recommendations on Health 
and Safety Regulatory Programs. 


yovernance of Crown corporations has 


improved but can be made better 


Corporate governance refers to the process and 
structures for overseeing the direction and management 
of a corporation so that it effectively carries out its 
mandate and objectives. It is critical that a Crown 
corporation be well governed if taxpayers’ money is to 
be well spent. Based on our general experience in 
auditing Crown corporations and a more specific 
review of 15 corporations, we see shortcomings in 
three areas important to the way Crown corporations 
are governed. 


First, boards of directors need strengthening. Currently, 
they lack key skills and capacities to oversee 
effectively the management of Crown corporations. 
Corporations need to define better their requirements in 
this area, and government needs to act on them. Boards 
of directors also need to be more engaged in selecting 
the corporation’s chief executive officer (CEO). 
Without meaningful involvement by the board in the 
process, accountability of the CEO to the board is 
weakened and corporate governance as a whole 
suffers. 


Second, the audit committees of Crown corporation 
boards often fall short of recognized best practices. 
The audit committee plays a crucial role in financial 
reporting, risk management and internal control. Of the 
committees we examined, half were considered 
ineffective or only marginally effective. 


Finally, the government has a limited capacity to 
review and challenge Crown corporation corporate 
plans. The government needs to strengthen this 
capacity, since these plans are the basis for holding 
Crown corporations accountable for their conformance 
to government policy and their performance overall. 


For related Auditor General work, see: 

2000 Chapter 18 Governance of Crown Corporations; 
1995 Chapter 10 Crown Corporations: Fulfilling 
Responsibilities for Governance. 


Report of the Auditor General of Canada — December 2000 


Matters of Special Importance — 2000 


Opportunities for cost savings 


Improving the efficiency and effectiveness of 
government activities and cultivating a healthy regard 
for economy are objectives central to our mandate and to 
the work that we do at the Office. The impact of our 
work is not easily quantifiable in most cases, but often in 
the course of our audits we find opportunities for cost 
savings to which a price tag can be attached. The 
following are examples taken from audits reported this 
year: 


Management of health benefits to First Nations. In a 
1997 audit of health programs delivered to First Nations 
communities, we noted the need for a point-of-service 
system to control the administration of benefits. We also 
recommended that Health Canada take appropriate steps 
to improve claims-processing system controls and edits. 
In a follow-up audit this year, we found that $8.4 million 
was saved in 1999 through identification and rejection of 
improper claims, but further improvements are still 
possible. 


Non-recovery of expenditures by Indian and 
Northern Affairs Canada. The Department has spent 
approximately $151 million since 1977 to ensure a 
continuous supply of drinking water to Cree 
communities that were adversely affected by 
hydro-electric development by Manitoba Hydro. Under 
an agreement signed in 1977, Manitoba Hydro is 
required to reimburse the Department for 50 percent of 
the expenditures attributable to the adverse effects of the 
hydro projects. We note in Chapter 17 that the 
Department has not made any recoveries to date. 


Debt servicing costs. In order to limit its exposure to 
unexpected increases in interest rates, the government 
increased the fixed-rate share of federal debt from 

55 percent of gross debt in 1995 to 67 percent in 1998, 
where it remains today. In Chapter 8 we recommended 
that in light of its improved fiscal condition the 
government review its debt structure target to determine 
whether it is still appropriate. The chapter noted that 
based on the historical difference between long- and 
short-term rates, a five-percentage-point shift from 
fixed-rate to floating-rate debt could generate an annual 
saving of some $250 million. 


For related Auditor General work, see: 2000 Chapter 8 
Department of Finance — Managing Canada’s Debt: 
Facing New Challenges; 2000 Chapter 15 Health 
Canada — First Nations Health: Follow-up; 2000 
Chapter 17 Other Audit Observations. 
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Service Quality 


Chapter 1 —- Main Points 


1.1 The purpose of our audit was to see whether the federal government is providing better quality of service 
to Canadians, after a decade of commitments and a series of initiatives to improve it. In 1996, we provided a 
midcourse assessment of the government’s progress. In the current audit, we revisited the same 13 services 
delivered by 10 government departments and agencies. 


1.2 The most significant improvement since 1996 has been to make telephone services more accessible. 
Canadians are now able to obtain faster responses to their enquiries once their calls are answered. Despite that 
improvement, however, we are concerned by the high percentage of calls not answered when lines are busy and 
calls abandoned while the caller waits on hold to speak to an agent. At the Canada Customs and Revenue Agency 
(Taxation) the percentage of unanswered calls remains high at 54 percent, although it has improved since 1996, 
when it was 73 percent. 


1.3 For most services delivered by means other than telephone, such as counter service, there was not enough 
performance information for us to determine whether service has improved. The Passport Office has measured 
some of its key results and demonstrated significant improvement in its performance. Many departments and 
agencies have taken initiatives since 1996 to improve service but have not sufficiently measured the results. 


1.4 Departments and agencies need to consult more with their clients to identify the aspects of service that 
matter most to them, as well as the quality of service they expect. This information would help the service 
providers establish appropriate targets and indicators of performance to measure. 


LS Communication to clients at points of service has improved since 1996. Canadians visiting these offices 
are better informed about the level of service they can expect. However, more attention is needed to informing 
them on what it costs to provide the services, whether targets have been met and how they can lodge and resolve 
complaints. 


Background and other observations 


1.6 The demand by Canadians for services of the federal government — such as enquiries about citizenship 
and immigration, Old Age Security, passports, weather or statistical information — has increased significantly 
since 1996. Volumes of service are higher in 10 of 13 service lines we audited. At the same time, the 
environment for delivering services continues to evolve with rapid advances in information technology and 
growing use of alternative methods of service delivery. 


Lf Telephone services have become increasingly prominent, while the use of counter services has declined. 
In the government telephone services we audited, the volume of telephone enquires climbed from 36 million in 
1996 to 56 million in 1999, an increase of 54 percent. The numbers of written enquiries (mail, fax and e-mail), 
while still important in some services lines, are generally much lower. 


1.8 Although service managers have acted on our 1996 recommendations and sought ways to make 
continuous improvements, none of the recommendations has been implemented fully. However, we noted a shift 
among public service managers and staff toward a stronger focus on service and innovation. 


1.9 We were discouraged to find slow progress on the project to improve the government’s telephone 
directory listings (the Blue Pages). The need to redesign them was identified in 1990 by the Public Service 2000 
Task Force on Service to the Public. 


nN 
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1.10 Over the past three years, departments and agencies have provided Parliament with more information on 
service quality, and the information they report is more likely to be meaningful. But better information is needed 


on performance trends and on costs. 


1.11 Since 1998, the Treasury Board Secretariat has given departments and agencies more guidance and 
support to improve service delivery. However, it has not systematically monitored progress across the government, 
and the information it has reported to Parliament has not been adequate to provide a clear understanding of 
progress in improving the quality of service to Canadians. 


The response of the Treasury Board Secretariat on behalf of the government is included at the end of the 
chapter. The Secretariat indicates that the new Blue Pages telephone directory format will be implemented 
according to established publication schedules. The Secretariat agrees with our recommendations on 
measurement, client satisfaction and reporting. It indicates that it is working with departments to develop 
an approach designed to promote continuous improvement in service delivery across the government. 


ee ee ee ee 
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Human Resources Development 
Canada 


service Quality at the Local Level 


Chapter 2 — Main Points 


2.1 Human Resources Development Canada (HRDC) has made considerable progress in addressing service 
quality. National initiatives in recent years have included developing a policy to guide its approach to high-quality 
service, implementing a new service delivery network to make services more accessible to Canadians and setting 
out national service commitments to clients. Initiatives at the local level include upgrading the service-related 
skills of employees and using new technologies. 


2.2 Although HRDC has defined national service commitments, we found they are not communicated 
consistently to clients at points of service. Further, local offices do not consistently inform clients about 
performance toward service delivery targets, the cost of services, and how to lodge complaints about the service 
received and obtain redress. 


2.3 The Department does not have performance measures for all of its national service commitments. 
Measures of service performance focus on waiting time for counter services and the speed of processing and 
payment, but there are weaknesses in the measurement of both these indicators. 


2.4 Despite improvements in the timeliness of Employment Insurance payments over the past three years, 
estimated incorrect payments have increased over this period from $4 to $6 for every $100 paid out. Available 
information shows that both underpayments and overpayments have increased for a variety of reasons, including a 
decline in the quality of information submitted by employers. We found that local office managers focussed 
mainly on speed of payment. There are neither performance targets nor measures of accuracy for the processing of 
claims at the local level. 


2.5 Employment assistance services are increasingly provided by non-governmental organizations, under 
contribution agreements with HRDC. The agreements we reviewed do not set out service performance 
expectations or targets. Nor do they require the non-governmental organizations to be accountable for the quality 
of the services they provide. 


2.6 There are gaps in the information the Department provides to Parliament on its performance in meeting 
its national service commitments. We also found gaps in the information that would help to provide context for 
the service performance that it does report. For example, information on the timeliness of paying Employment 
Insurance claims cannot be assessed in relation to information on the accuracy of payments. 


Background and other observations 


2.7 Almost one in three Canadians is a client of HRDC. Each year the Department processes 60 million to 
80 million program claims and applications, sends about 100 million letters and pays out over $50 billion in 
various benefits. Most programs and services are offered at the local level, through one of the federal 
government’s largest service delivery networks. It comprises 320 Human Resource Centres of Canada, 

5,000 self-service kiosks, 21 telephone centres and 10 mail centres. Almost 80 percent of HRDC’s 

21,000 employees work at the local level. 


2.8 To make continuous improvements in the quality of services, managers need to seek more feedback from 
clients at the local level through such means as client satisfaction surveys and analysis of client complaints. There 
is no policy for handling complaints in either the Employment Insurance Program or the Income Security 
Programs. However, the latter has started to develop such a policy. 
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2.9 The Department does not have the cost information it needs to manage the trade-off between the cost of 
its main services and their quality. We believe it needs to give more attention to developing and implementing 
action plans at the local level for continuous improvement in service quality. 


The Department’s responses to our recommendations are included in this chapter. For the most part, the 
Department agrees with the recommendations and indicates the steps that it is taking or intends to take to 
address them. 
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Citizenship and Immigration 
Canada 


The Economic Component of the 
Canadian Immigration Program 


Chapter 3 - Main Points 


3.1 We noted serious deficiencies in the management and delivery of the economic component of the 
Canadian Immigration Program, whose purpose is to recruit highly qualified individuals who can readily 
contribute to our economy and adapt to our society. These deficiencies seriously limit Canada’s ability to 
maximize the economic and social benefits that immigration affords, and to protect the integrity of its 
Immigration Program. 


ee We found that immigration offices abroad are overtasked. They have much difficulty coping with the 
volumes of work and responsibilities assigned to them. Immigration levels set by the government are not met, and 
applicants are waiting longer and longer for their applications to be finalized — sometimes more than three years. 


3.3 We noted weaknesses that leave Citizenship and Immigration Canada open to criticism of the quality and 
consistency of its decisions in selecting immigrants. Visa officers need better selection criteria, better training and 
better tools to assess immigrant applications more effectively. In addition, the Department does not sufficiently 
monitor the quality of their decisions. We noted significant differences in approval rates among officers processing 
similar cases. 


3.4 Our audit revealed significant weaknesses in the management of medical assessments of prospective 
immigrants. Since our last audit in 1990, the Department and Health Canada have been unable to take a position 
on whether changes to standards for medical examinations are required to determine if an applicant poses a danger 
to public health and safety, or could place excessive demand on health care systems or social services. The same 
routine tests have been required for the last 40 years and the departments have not decided whether applicants 
should be routinely tested for any infectious diseases other than syphilis and tuberculosis. We are also very 
concerned about the lack of rigour and consistency in the overall management of medical assessment activities, 
including the procedures for supervising the designated local physicians who perform medical examinations of 
prospective immigrants abroad. 


a There are serious constraints on establishing criminality and security admissibility of prospective 
immigrants. Visa officers have little information and support to ensure that applicants are not likely to engage in 
criminal activities or endanger the safety of Canadians. Some people are thus admitted to Canada without 
reasonable assurance that they have not committed crimes abroad, engaged in espionage, subversion or terrorism, 
or been associated with organized crime. 


3.6 The Department is particularly vulnerable to fraud and abuse. It has no effective measures in place to 
discourage people from submitting fraudulent applications, and visa officers often resort to detection methods that 
are costly. In addition, we found inadequate control over revenues, visa forms and computer systems. 


aud Employees responsible for processing immigrant applications in offices abroad are deeply concerned 
about the present state of affairs. They feel they are not only going against their own values but also making 
decisions that could carry risks that are too high and that could entail significant costs for Canadian society. 


3.8 Overall, it is highly questionable whether the Department has the resources and the operational capacity 
it needs to carry out the tasks required to meet the annual immigration levels set by the government. 
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Background and other observations 


RA Citizenship and Immigration Canada is responsible for, among other things, setting immigration policy, 
selecting immigrants, and evaluating visitors’ applications abroad. It also protects Canadians’ health and safety by 
intercepting individuals who are not entitled to enter or remain in Canada, and returning them to their country of 


origin. 


3.10 In 1999, approximately 190,000 immigrants were admitted to Canada. There are four broad components 
in the Immigration Program: family, economic, refugee, and other. Our audit focussed on the economic 
component, which accounts for 56 percent of landed immigrants each year. This component is aimed at recruiting 
skilled workers and business immigrants. 


3.11 Immigration services abroad are provided through a complex network of 81 offices in Canadian 
embassies, high commissions and consulates. About 210 Canadian officers and 980 locally engaged staff work in 
these offices and process a large volume of immigration visas, visitor visas, and student and temporary 
employment authorizations each year. Several other federal departments and agencies also have responsibility for 
results achieved in the Immigration Program, including the Department of Foreign Affairs and International 
Trade, the Canadian Security Intelligence Service, the Royal Canadian Mounted Police and Health Canada. 


3.12 It is disappointing to note that several of the deficiencies discussed in this chapter are similar to those we 
reported in 1990. Over the years, the government and the Department have undertaken a number of initiatives to 
remedy the situation, but without much success. 


3.13 This situation requires immediate attention by both the Department and the government. On the one 
hand, the Department must improve its efficiency in processing immigrant applications. Offices abroad are buried 
in paperwork, and their outdated technology is a serious obstacle to improving performance. The Department 
needs much better information on the resources required to fulfil its responsibilities in processing applications. On 
the other hand, a significant investment of resources may be needed to correct the deficiencies we noted and to 
allow the Department to realize the potential efficiency gains we identified. It is particularly essential that in 
deciding on annual immigration levels, the government ensure a proper balance between the resources allocated to 
the Department and its federal partners and the workload required to meet those levels. 


3.14 The Department’s current review of the Immigration Act provides an excellent opportunity to correct 
some of the deficiencies. However, the challenge for the government and the Department is enormous and 
complex. Since Parliament has expressed a particular interest in the Program by requiring the tabling of an annual 
immigration plan, we encourage parliamentarians to follow the situation closely. 


The responses of Citizenship and Immigration Canada, Health Canada, the Royal Canadian Mounted 
Police, the Canadian Security Intelligence Service and the Department of Foreign Affairs and International 


Trade are included in this chapter. They agree with our recommendations and note that initiatives are under 
way in several areas. 
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Indian and Northern Affairs 
Canada 


Elementary and Secondary 
Education 


Chapter 4 - Main Points 


4.1 Indian and Northern Affairs Canada cannot demonstrate that it meets its stated objective to assist First 
Nations students living on reserves in achieving their educational needs and aspirations. For example, the 
Department does not have the necessary assurance that First Nations students are receiving culturally appropriate 
education. Moreover, the progress in closing the education gap for Indian students living on reserves has been 
unacceptably slow. At the current rate of progress, it will take over 20 years for them to reach parity in academic 
achievement with other Canadians. 


4.2 To obtain assurance and effectively discharge its responsibilities, the Department needs to resolve several 
major issues. These include the need to articulate its role in education, to develop and use appropriate 
performance measures and to improve operational performance. In addressing these issues, the Department will 
need to further take into account the cultural and special needs of Indian students as well as socio-economic 
factors that can affect success in education. 


4.3 The situation is complex and urgent, and the urgency will increase as more demands are placed on 
education as a result of an increasing population on reserves, and changes in technology and in provincial 
education programs upon which the Department relies. Although the Department has identified or started several 
projects for education reform, it must still respond with significant action to resolve these issues. 


Background and other observations 


4.4 The Department is empowered to take an active role in the provision of elementary and secondary 
education for Indian children living on reserves. Although it has chosen to rely on First Nations and the provinces 
for the design and delivery of appropriate education, the Department acknowledges that this approach does not 
diminish its responsibility and accountability. 


4.5 For more than 100 years, various statutes, Indian treaties and government policies have shaped the 
mandate and responsibilities of Indian and Northern Affairs Canada for education. 


4.6 Presently, about 117,000 students enrolled in elementary and secondary schools live on reserves. Current 
budgeted costs, not including school construction and maintenance, are about $1 billion annually. 


Indian and Northern Affairs Canada acknowledges the importance of the issues identified in this chapter. It 
believes that Gathering Strength — Canada’s Aboriginal Action Plan, and other planned education 
initiatives involving the Department, First Nations and provincial governments, will help close the education 
gap cited in this chapter. 


Report of the Auditor General of Canada — April 2000 


SES 


Canada Customs and 
Revenue Agency 


Travellers to Canada: 
Managing the Risks at Ports of Entry 


Chapter 5 — Main Points 


A | The Canada Customs and Revenue Agency has many efforts under way to modernize its Customs 
operations and to make effective use of technology. However, we found that it needs to make some significant 
improvements. 


5.2 A well-articulated and comprehensive compliance strategy is a key element of promoting compliance. 
Customs does not yet have an overall compliance strategy that clearly describes its plans for encouraging 
voluntary compliance and that outlines its approach to responsible enforcement. 


5.3 We found that risk assessment is incomplete: Customs does not have important information it needs from 
a variety of departments and agencies to fully assess the risks its inspectors face. It needs to know where the risks 
are highest so it can determine the best way to control them. We have recommended that Customs work more 
diligently to obtain information on the risks arising from the responsibilities it carries out at ports of entry on 
behalf of other departments — Citizenship and Immigration Canada, Health Canada and the Canadian Food 
Inspection Agency, for example — and include them in its national risk assessment. It also needs to have 
up-to-date memoranda of understanding with those departments, setting out their respective roles and 
responsibilities. 


5.4 We also found that Customs inspectors at the border and at airports need to be better equipped to do their 
complex job. They must enforce customs and excise legislation and make a variety of decisions for many other 
government departments. In processing travellers, Customs inspectors have only a short time to make these 
decisions. Customs does have information, systems and training to assist its officers but the information is not 
shared consistently, the systems need improvement and the training is uneven. 


Background and other observations 


th) In 1998-99 the Customs program of the Canada Customs and Revenue Agency processed over 

104 million travellers who entered Canada at ports of entry. About half were returning Canadian residents. Most 
travellers enter Canada by highway from the United States (79 percent) or at airports (17 percent). Since 1991-92 
the number of travellers coming by highway has decreased, while the number of air travellers has grown by 
almost 48 percent, owing partly to the Canada-United States Open Skies Agreement. 


5.6 Over the years, Customs’ role has evolved from mainly collecting revenue to one of facilitating the entry 


of travellers and goods, while protecting Canadian society and promoting the competitiveness of Canadian 
businesses. 


5.7 Customs operates in an environment that continues to change. To help it deal with change, it developed 
discussion papers in 1990 and again in 1998 as blueprints for the direction it would take in the future and the 
initiatives it would employ. Customs has taken some major steps toward reaching the goals it set for itself in 1990. 


It recently released a draft five-year action plan to implement the initiatives identified in its 1998 discussion 
paper. 


The Canada Customs and Revenue Agency’s responses to our recommendations are included in the chapter. 


The Agency agrees with the seven recommendations, and its responses describe a number of actions under 
way to deal with them. 


12 
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Canada Customs and Revenue 
Agency and Department of Finance 


Handling Tax Credit Claims for 
scientific Research and 
Experimental Development 


Chapter 6 — Main Points 


6.1 In 1994 the government restricted the period for filing tax credit claims for scientific research and 
experimental development (SR&ED) to 18 months after the year in which taxpayers incurred them. The Canada 
Customs and Revenue Agency then received 16,000 additional claims, which represented $2.8 billion in credits 
for taxation years as far back as 1985. 


6.2 Our audit revealed significant inconsistencies in the handling of these claims. These inconsistencies 
compromised the fairness of the review process. In addition, serious professional differences of opinion arose 
within the Agency on how to resolve a dispute with a claimant over a significant claim. The Agency did not 
demonstrate that it had effective mechanisms to resolve this type of difference among its officers on individual 
cases. 


6.3 The lack of clarity about the eligibility of SR&ED projects has resulted in unresolved claims dating back 
to 1985. The claimed tax credits amount to hundreds of millions of dollars. Since the creation of the SR&ED 
program, there have been disagreements between claimants and administrators and among administrators on what 
constitutes an eligible activity or project, how thoroughly a science officer should review a claim and how much 
documentation is necessary to support a claim. Clearer rules are urgently needed to deliver and administer the 
program effectively and efficiently. 


6.4 The Agency has recognized the need to improve the administration of the program. To this end, after 
consulting stakeholders, the Minister of National Revenue released an action plan in November 1998. The Agency 
created a new Directorate and appointed a new head of the program. 


6.5 According to our audit, the Agency needs to improve the assessment of the risk that taxpayers’ claims are 
ineligible so that science and audit staff can focus on claims with the highest risk. The Agency manages the risk of 
ineligibility by conducting completeness checks, risk assessments, audits and science reviews. 


6.6 An evaluation of the SR&ED program, done by the Department of Finance and the Agency, may have 
overstated the impact of the program in generating additional spending on SR& ED. The evaluation also indicated 
that the program generated only a net increase in Canada’s real income of between $20 million and $55 million 
per year. 


Background and other observations 


6.7 The federal government encourages research and development in the private sector through tax 
incentives. Taxpayers receive tax credits once they have incurred expenses on their projects. 


6.8 The Agency’s information systems show that in 1997 about 11,000 corporations and individual taxpayers 
claimed over $10 billion in SR&ED expenses and $2 billion in tax credits. Large corporations claimed 85 percent 
of total expenses. 

6.9 A staff of 144 science advisors and 268 financial auditors review claims in 37 tax services offices across 
the country. Advisors determine whether the work is eligible under the program and auditors examine whether the 
related costs qualify. 
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6.10 A-key objective of the Agency’s 1998 action plan was to reduce uncertainty about the eligibility of 
science projects. As a result, the Agency and industry proposed a definition of an SR&ED project that, in our 
view, modifies the level of eligible SR&ED work and expenses and that could result in additional costs in tax 
credits. If any significant changes were made to the definition of an SR&ED project, they would need to be 
reviewed by the Department of Finance, which is responsible for the program’s policy objectives, costs and related 
legislation. 

The responses of the Canada Customs and Revenue Agency and the Department of Finance are included in 
the chapter. Their responses describe the action they have taken and will continue to take to address our 
recommendations. 
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Royal Canadian Mounted Police 


Services for Canada’s Law 
Enforcement Community 


Chapter 7 — Main Points 


jm The levels of service that the RCMP provides to the Canadian law enforcement community do not meet 
the needs of its clients. For example, the Canadian Police Information Centre (CPIC), a backbone system that 
provides information to police officers, courts, prisons and parole officers, was unavailable regionally or 
nationally 11 percent of the time last year. DNA analysis takes too long and is limited to only the most important 
cases. Thus, its full potential for public safety is not being realized. Backlogs of up to two months exist for 
inputting criminal history records and fingerprints into data banks. As a consequence, the criminal records and 
fingerprint data available to the police and others are not current. In addition, criminal record checks required by 
employers or volunteer organizations take too long. 


7.2 It is imperative that senior management at the RCMP take action to eliminate backlogs in many of the 
services and improve efficiency. In particular, the RCMP needs to rationalize the number of laboratories in order 
to improve their level of service. Significant improvement is also required in performance measurement and the 
reporting of results. It is important that users, senior RCMP management and Parliament know these results. 


on The arrangement between the federal government and the provinces set in 1966 for these services needs 
to be rethought. It is time for a clear agreement among all the players in the law enforcement community — in the 
federal, provincial and municipal governments — on level of service, funding arrangements, user input, 
management and accountability. A new agreement will require the collaboration of all parties. 


Background and other observations 


7.4 The RCMP provides a range of services that are essential to the Canadian law enforcement community: 
police, courts, prisons and parole boards. These services include fingerprint identification, criminal history 
records, forensic laboratories, criminal intelligence and on-line access to data banks such as vehicle registration, 
drivers’ licences, wanted persons and stolen property. In 1998-99, the RCMP spent approximately $122 million 
on these services. 


72 There are a number of areas where the RCMP deserves credit for the way it has operated these services: it 
has kept the old CPIC information system operational while awaiting a new system; it has implemented new 
forensic techniques such as DNA analysis; it is developing a data bank for DNA; and it has made major upgrades 
to fingerprint identification and criminal history records systems. 


7.6 Our last audit of services to the law enforcement community was in 1990. We made a number of 
recommendations to improve operational economy and efficiency. Attention has only recently been paid to many 
of our recommendations and others have not been addressed. These national services have also been reviewed 
several times over the past decade by the Solicitor General Canada, the Treasury Board Secretariat and the RCMP 
itself. 


The RCMP agrees with the findings and indicates that it will be taking action. 
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Le eS 


Department of Finance 


Managing Canada’s Debt: 
Facing New Challenges 


Chapter 8 — Main Points 


8.1 Canada’s debt management program is a well-run operation overall. But like any program that faces a 
changing environment, it needs to adjust. We have noted three areas where the Department of Finance needs to 
review its practices. These include: 


* establishing a more complete range of performance measures to ensure that it has a sound basis for 
assessing how well the program is doing at meeting its objectives. Currently, the Department uses a 
variety of information internally but reports its performance for only one target — the ratio of 
fixed-rate debt to the total debt, now at two thirds; 


© improving transparency to ensure that that there is a public accounting of not only the benefits of 
decisions on all aspects of debt management but also the costs. In recent years, the government 
significantly increased the proportion of fixed-rate debt and the level of foreign exchange reserves, 
and established targets for retail debt — all without disclosing the expected costs of these decisions; 
and 


¢ modifying the management of strategic planning by consulting a range of outside experts to ensure 
that the Department obtains complete and independent advice. 


Background and other observations 


8.2 At the end of 1998-99, Canada’s interest-bearing debt stood at $595 billion and the annual interest 
charges on that debt amounted to $41.4 billion. Managing the debt is the responsibility of the Department of 
Finance, working in conjunction with the Bank of Canada and the Canada Investment and Savings agency. 


8.3 The principal objective of the debt management program is to provide stable, low-cost funding for 
government operations while keeping liquid the domestic markets for that debt. Another objective is to ensure that 


there are adequate reserves in the Exchange Fund Account to moderate volatility in the exchange rate on the 
Canadian dollar. 


8.4 We found that the people who manage this program are a highly committed group. They closely monitor 
fiscal and economic developments that could affect debt management operations, and they consult regularly with 


financial market participants. They use sophisticated modelling techniques to identify debt management risks and 
to choose appropriate debt strategies. 


8.5 Our audit dealt not so much with the choices debt managers make or the analysis underlying those 


choices as with the way they measure performance, the information they report, and the governance of the 
program. 


8.6 . Over the last few years, the government has made some major decisions in managing the federal debt. It 
has increased the proportion of longer-term, fixed-rate debt to two thirds of the total, some 18 percent higher than 
three years ago. While fixed-rate, longer-term debt makes the government’s interest costs more predictable and 


reduces rollover risk, it generally costs more because long-term interest rates are normally higher than short-term 
rates. 
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8.7 Over the same three-year period, Canada’s foreign exchange reserves have more than doubled, to 
US$25 billion. A higher level of foreign reserves improves the government’s ability to promote stability in the 
value of the Canadian dollar, and also provides a larger pool of funds that can be used to finance unanticipated 
needs. But this comes at a cost, because the interest paid on foreign currency borrowing is normally higher than 
the interest earned on foreign reserve assets. 


8.8 We also looked at retail debt — the sale of Canada Savings Bonds and Canada Premium Bonds. We 
noted that despite renewed attention to the sale of these bonds and the creation of a special operating agency to 
market them aggressively, we have seen no conclusive evidence that retail debt is cost-effective. We encourage 
the government to review the role of the retail debt program and its cost implications. We also encourage the 
government to disclose the results of this review when it completes it. 


The Department’s response to our recommendations are included in this chapter. For the most part, the 
Department accepts the thrust of our recommendations and indicates how it is responding to them. 
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SSS 


Streamlining the Human 
Resource Management Regime 


A Study of Changing Roles 
and Responsibilities 


Chapter 9 - Main Points 


ii! We undertook this study to draw to Parliament’s attention the urgent need to deal with long-standing 
issues and emerging challenges for human resource management in the public service. 


9.2 The current framework governing human resource management in the “core” public service is unduly 
complex and outdated. Administrative systems are cumbersome, costly and outmoded. The framework is ill suited 
to an environment that demands flexibility and adaptability — an environment that faces significant challenges in 
human resource management and an increasingly competitive labour market. 


9.3 Public service staffing is a major source of frustration both to managers and to employees. The system is 
rule-bound and inefficient. Managers need to have more authority in staffing, but they also must be more clearly 
accountable for their decisions. The interests of employees must be respected, but there is a pressing need to 
modernize and streamline the processes for staffing and related recourse. 


9.4 Concerns about “fractured responsibility” for human resource management are long-standing. 
Responsibility and accountability for the changes needed to simplify, streamline and strengthen the current human 
resource management regime need to be clearly assigned and appropriately supported. This is particularly 
important in areas of divided responsibility. 


¢ In human resource management, deputy ministers — leaders of the major organizations of 
government — have been seen as primarily responsible for administering a centrally prescribed 
framework. A significant step in their role is needed. Deputies should have pivotal responsibility for 
developing and maintaining a healthy work environment in their departments, and their 
responsibilities for human resource management need to be set out formally. Deputy ministers should 
be clearly accountable for the way their departments perform these responsibilities. 


The Public Service Commission needs to engage in an active dialogue with Parliament about 
changes in its activities and in the way it carries out its responsibilities for merit protection. This 
dialogue should encompass legislative reform of staffing. The Commission also needs to improve its 


reporting on departmental performance in adhering to the provisions and principles of the Public 
Service Employment Act. 


There is a need for the Treasury Board to strengthen its reporting to Parliament on the aspects of 
human resource management for which it is responsible. As well, there is a need to clarify the 


responsibility of deputies for reporting on the quality and effectiveness of human resource 
management in their departments. 


Background and other observations 


9.5 The core public service is now substantially smaller than at any time since the early 1970s. It has 
diminished by almost 100,000 employees in the last 10 years, to about 143,000 employees by the end of 1999. 
Government downsizing, devolution, privatization, limited recruitment and other measures were factors in this 
reduction. More than 50,000 public servants have moved from the “core” public service (for which Treasury 
Board acts as the employer) to “separate employers” like the Canada Customs and Revenue Agency. 
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9.6 Knowledge workers now constitute 55 percent of public servants (up from about 33 percent 15 years 
ago). They are bringing new expectations to the workplace, heightening the need for change. 


9.7 The percentage of public service employees aged 45 to 54 has almost doubled over the last 14 years, and 
70 percent of executives could retire within 10 years. This is generating concern about a potential leadership 
crisis. Moreover, youth are underrepresented: the percentage of public servants under 35 years of age is roughly 
half that in the Canadian work force. 


9.8 In 1996, the Treasury Board Secretariat estimated that there were 840 separate pay rates and 70,000 rules 
governing pay and benefits. In 1997, there were more than 12,000 pages of instructions in the Treasury Board’s 
personnel and pay administration manuals. It takes 119 calendar days on average to complete a closed competition 
in the core public service, not including added time to deal with any appeals. This is about twice as long as 
reported by selected quasi-public organizations. For a new position that needs to be classified, staffing takes 

230 calendar days on average — almost eight months. 


In its corporate response, the government agrees on the need for a strong human resource management 
framework and on the importance of the health of the public service. It is more optimistic than the Auditor 
General on some matters, and notes recent initiatives to address some issues. In a supplementary response, 
the Public Service Commission indicated its continuing efforts to improve staffing systems and to engage in 
a dialogue with Parliament. 
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Transport Canada 


Airport Transfers: National Airports System 


Chapter 10 - Main Points 


10.1 Our audit examined how Transport Canada handled the transfers of Canada’s largest and busiest airports 
between 1992 and 1999. These airports make up Canada’s National Airports System. Under the transfer 
agreements, Transport Canada retains ownership of the airports but leases out their management, operation and 
development to bodies known as airport authorities. Our audit concentrated on the financial and oversight aspects 
of airport transfers in the National Airports System (NAS), not on security and safety. 


10.2 We found many significant weaknesses in management practices. Among our most important 
observations are the following: 


e Before it started the lease negotiations for each airport transferred in the second round, and any 
renegotiations, Transport Canada did not determine the fair market value of the airport assets and 
business opportunities it was transferring. Such information is fundamental to both negotiating and 
renegotiating leases and, in our view, its absence represents a clear departure from sound 
management practice. The quality of information for making decisions on such things as rent is 
significantly impaired as a result. 


e The Department has renegotiated four leases, at a cost to the government of about $474 million in 
forgone rent ($342 million net present value). The renegotiated deals do not adhere to some of the 
government’s key directions. Further, Transport Canada cannot demonstrate how the deals for all of 
the transferred airports are equitable, uniform, consistent and fair, one with the other, as the 
government directed. 


e As aresult of renegotiations, the government has, in effect, agreed to a reduction of future revenues 
of the Crown and to the funding of significant capital projects. The information presented to 


Parliament on forgone rent and the funding of capital projects was fragmented, incomplete and, in 
some years, non-existent. 


e From 1992 to 1999, the government continued to provide financial support to most of the transferred 
airports. It provided to the airport authorities a total of $360 million, including $118 million in rental 
credits at Lester B. Pearson International Airport toward a number of renovation projects. The 
government received a net total of $593 million in rent from airport authorities in this period. 


e As assessed by its consultants, the Department’s preliminary financial results indicated that 
five years after transferring the first four airports in 1992, the government’s most likely financial 
position varied significantly after each transfer — from better off to worse off. Although the analysis 
had been completed — as part of three separate studies with a total cost of $680,000 — a year before 
our audit ended, we found that Treasury Board and Cabinet had not yet seen the results. The 
Department has yet to conduct any such analysis for any of the other transferred NAS airports. 


We are concerned that eight years into the transfer process, Transport Canada has yet to clearly 

deine its role as landlord and overseer of the National Airports System. Its handling of key emerging 
issues such as those related to airport improvement fees, subsidiaries and sole-source contracting has 
generally been inadequate and, until 1997, was virtually non-existent. Treasury Board and/or Cabinet 


have received little information on these issues, and some of what they have received has been 
incomplete and inaccurate. 
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Background and other observations 


10.3 Airports in the National Airports System have been transferred in two rounds. In 1992, Transport Canada 
leased out the management and administration of four major airports — at Montreal (Dorval and Mirabel), 
Vancouver, Calgary and Edmonton — to four “local airport authorities”. The second round began in 1996 with the 
transfer of Toronto’s Lester B. Pearson International Airport. Since then, 12 other airports, including those at 
Ottawa, Victoria, Winnipeg and Moncton, have been transferred to “Canadian airport authorities”. At the end of 
our audit, the last four NAS airports — Gander International, Québec/Jean-Lesage International, Fredericton and 
Prince George — remained to be transferred. 


10.4 Although Transport Canada began transferring airports in the late 1980s and has been leasing out airports 
since 1992, only in 1994 was the National Airports Policy issued. The government saw transfers as a means of 
funding expansion in the vital National Airports System, making airports more competitive and viable and giving 
communities the flexibility to use them as tools for economic development. At the same time, Transport Canada 
would be able to oversee the entire System. 


10.5 Transport Canada notes that a number of aspects of the transfer initiative have been positive and that 
airport authorities have made some strategic choices that have also represented difficult operational decisions. 
These included, for example, the expansion of passenger facilities, liberalization of operating policies and 
relocation of scheduled traffic from one airport to another. 


The Department’s responses to our recommendations are included in this chapter. While the Department 
agrees with the majority of the recommendations and indicates the steps that it is taking or intends to take 
to address them, it takes a different position on a number of issues as reflected in its response following 
paragraph 10.106. 
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Human Resources Development 
Canada 


Grants and Contributions 


Chapter 11 — Main Points 


11.1 Our audit examined four of the grant and contribution programs included by Human Resources 
Development Canada (HRDC) in its 1999 internal audit. We concluded that there were widespread deficiencies in 
the management control frameworks of all four programs. Our findings, which covered the period to December 
1999, confirmed — and extended — those of the Department’s 1999 internal audit. Among other things, we found 
breaches of authority, payments made improperly, very limited monitoring of finances and activities, and 
approvals not based on established processes. 


11.2 The results that projects were to achieve were often not defined in terms that could be measured. Even 
when they were (as in the case of Youth Internship Canada), results were not measured systematically to provide 
managers with feedback for making necessary improvements and to provide a basis for accountability. Some 
evaluation of these programs had been done. The evaluations of the Sectoral Partnerships Initiative and 
Transitional Jobs Fund resulted in some program changes. However, we could not support the Department‘s 
findings on the effectiveness of the Transitional Jobs Fund. 


11.3 The Department’s review of 37 “problem files” identified by its internal audit for further follow-up 
established few cases where money could be recovered from project sponsors, because most payment errors 
resulted from inappropriate practices by departmental staff. Many of these practices are not acceptable — 
proceeding without required approvals, relying on oral contracts and paying for ineligible expenses, for example. 


Reporting by HRDC on the results of the review focussed on overpayments and provided little information on the 
problems found. 


11.4 Current management is committed to addressing the serious weaknesses in the management control 
framework in the period up to December 1999 that our audit and the 1999 internal audit identified. HRDC has 
corrective action planned and being implemented in response to the problems identified in its 1999 internal audit. 
The actions and plans also address the deficiencies we found in our audit. 


11.5 The Department has made good progress toward meeting the commitments in its Six-Point Action Plan 
announced in February 2000. Management is enhancing the tools and the support available to staff to improve 


their ability to do their jobs. Work is also proceeding on additional initiatives that expand or complement the 
original action plan. 


11.6 HRDC management will need to sustain its efforts and attention if it is to achieve the broad-based 
systemic change the action plans envision. Beyond the immediate corrective steps the Department has taken, it 
needs to make today’s extraordinary effort tomorrow’s routine and fundamentally change its day-to-day approach 
to the delivery of grants and contributions. Management and staff of the Department need to continue the current 
focus on the fundamentals of control. Minimum standards of control must be in place for all projects. However, 
time and effort needed to review and assess proposals, monitor progress, assess performance, and evaluate results 
should vary with the amount of federal funding involved and the risks associated with the project. 


11.7 HRDC has established an innovative system of tracking performance — one that allows for tracking 
improvements in the management and administration of grants and contributions. Ongoing performance tracking 
and internal audit are essential tools for assessing improvements in the management control framework. 
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Background and other observations 


11.8 HRDC spent about $3 billion in 1999-2000 on grants and contributions for programs such as job creation 
and youth employment, as well as for employment benefits and support measures. We examined in detail four of 
about 40 grant and contribution programs run by HRDC: the Transitional Jobs Fund and its successor the Canada 
Jobs Fund, Youth Internship Canada, Social Development Partnerships and the Sectoral Partnerships Initiative. 


11.9 HRDC’s problems in managing grants and contributions worsened in the 1990s; audits in the late 1990s 
show the persistence of problems identified in the past. An internal audit in 1991 and its follow-up in 1994 led to 
little corrective action. Management’s priorities then were to implement major policy initiatives and improve 
service; there was not enough emphasis on maintaining essential controls while red tape was being reduced and 
service improved. 


11.10 More work is required to determine the resources needed to deliver grant and contribution programs. The 
Department’s corrective actions and plans are designed to strengthen capacity by providing staff with appropriate 
support, training, tools and systems. HRDC is also taking steps to bring in new staff at appropriate levels. It plans 
to analyze workload to ensure that it has adequate resources in place. 


11.11 The programs we audited had made available general information about their existence. For the most 
part, the promotion of these programs was passive and, in the case of some programs delivered at the local level, 
varied significantly among regions and local offices. Information was not always provided in a way that would 
promote equal access to the programs. 


The chapter includes the Department’s general response and its responses to specific recommendations. 


HRDC has responded positively to our findings and recommendations and affirmed that it will continue to 
sustain the progress shown to date in meeting the commitments of the Action Plan and related initiatives. 
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Values and Ethics in the 
Federal Public Sector 


Chapter 12 - Main Points 


12.1 Canadians expect that the federal public sector will be a world leader in promoting sound values and 
ethics in government. Ensuring sound values and ethics is a vital part of good governance that supports and 
respects fundamental democratic values. 


12.2 A comprehensive and sustained dialogue with Canadians and throughout the federal public sector is 
required. We propose a framework for action that includes eight priorities. We suggest that action start with the 
following two priorities. 


e Reinforce leadership for promoting ethical conduct by, for example, 
- clarifying the principle of ministerial responsibility and the responsibilities of officials; and 


- identifying the responsibilities of senior and line managers for promoting sound values and 
ethics. 


e Re-invigorate an extensive dialogue on values and ethics that emphasizes the primacy of the 
principles of respect for the law, the public interest, and public service as a public trust. 


12.3 Sound values and ethics are needed to maintain probity. There is a strong foundation of values and ethics 
in the federal public sector. This foundation should not be taken for granted. The government is taking steps to 
maintain sound values and ethics. Most of these measures are limited or in their preliminary stages. To increase 
the likelihood of success, values and ethics initiatives have to be carefully developed, implemented, and 


monitored by using the best available practices. A prerequisite for success is the leadership of parliamentarians, 
ministers and senior officials. 


12.4 The federal government needs to develop an implementation plan with deadlines to address values and 
ethics priorities across the public sector and in federal entities. We will continue to monitor and report to 
Parliament on values and ethics initiatives in the public sector. 


Background and other observations 


12.5 We focussed on values and ethics initiatives that promote ethical decision making that is in the public 
interest. These initiatives would form part of a comprehensive approach to good governance in federal entities. 


12.6 In May 1995 we reported to Parliament on ethics and fraud awareness in government. In 1996 the 
government completed A Strong Foundation: Report of the Task Force on Public Service Values and Ethics 


(Tait Report). These reports contributed to forging a consensus on the values and ethics issues that need to be 
addressed in the federal public service. 


12.7 Ensuring ethical decision making is the responsibility of all members of the federal public sector and 


private sector companies as well as organizations and individuals who receive funds from or do business with the 
public sector. 


12.8 Major changes are occurring in the way government organizes and delivers its programs. These changes 
include deregulation, downsizing, increasing delegation of decision making to officials in the field, contracting 
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out and entering into partnerships with the private sector. As well, the government is using new technologies, and 
its work force is becoming more mobile and diverse. As a result, the government is increasingly relying on a 
strong foundation of values and ethics to make decisions in the public interest. 


12.9 As part of good governance efforts, several departments are implementing major values and ethics 
initiatives. Others do not assign a high priority to developing comprehensive values and ethics initiatives. We 
believe that all entities should objectively assess and report on values and ethics. 


The responses of the Privy Council Office, the deputy minister co-champions for values and ethics in the 
public service, the Secretary of the Treasury Board and the Ethics Counsellor are included at the end of this 
chapter. They agree with our assessment of the initiatives under way and acknowledge that there is an 
important need for further work. 
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Assessment of Financial Management 
Capabilities in Departments 


Chapter 13 - Main Points 


13.1 Our audit assessed the financial management capabilities in five government departments against the 
criteria established in the Financial Management Capability Model published in April 1999. We found that in each 
department gaps remain between current expectations for financial management and the existing capabilities of 
the departments. The capabilities needed to meet current expectations are consistent with those we have described 
in the Control Level of the Financial Management Capability Model. 


13.2. We also noted that departments are at the initial stages of developing the capabilities needed to meet the 
requirements of the government’s Financial Information Strategy and the Modernization of Comptrollership 
initiative. These capabilities are consistent with the capabilities described in the Information Level of the 
Financial Management Capability Model. 


13.3 Based on the level of financial management capabilities in the departments that we audited, the 
challenges to be met in improving financial management are substantial. Only one department is close to meeting 
all the government’s current expectations and none currently have the capabilities needed to meet the goals of the 
Financial Information Strategy and Modernization of Comptrollership initiative. Put simply, it will be very 
difficult to make government more businesslike if stronger financial information is not well integrated into both 
day-to-day management and accountability reporting to Parliament. The achievement of these capabilities 
throughout government will require the strong commitment and sustained support of the Treasury Board 
Secretariat, Privy Council Office and deputy ministers. 


13.4 Our Office has expressed concern about the need to improve financial management in government for a 
number of years. Similarly, the Public Accounts Committee has noted, “There has been a common call for 
improvements in financial information to support government decision making.” We have noted an increase in the 
number of initiatives being undertaken and a heightened sense of urgency to develop strong financial management 
capabilities. The government and the departments themselves have much to do in this area. The success of Results 
for Canadians — A Management Framework for the Government of Canada, the Financial Information Strategy 
and the Modernization of Comptrollership initiative will depend on this action. 


Background and other observations 


13.5 Financial management is an important component of what financial and program managers in 
departments and agencies do in delivering programs and services and exercising stewardship over the resources 
entrusted to them. An integrated and systematic approach to financial management allows managers to have the 
information to make sound decisions, manage risks well, and account properly for the use of public resources. 
Until recently, financial management has focussed on controlling budgets and processing transactions. Today’s 
rapidly changing environment makes it urgent for the government to implement effective financial management. 


13.6 We noted that departments had not implemented adequate monitoring of the effectiveness of their control 
frameworks. Such monitoring, together with providing assurance to senior management that control mechanisms 
are operating as designed, is an important component of sound management, particularly in a period of significant 
change that accompanies the implementation of new financial systems. 


13.7 One of the most common gaps that we found in departments was the limited ability to combine or 
integrate financial and operational (non-financial) information. This gap has clear implications for individual 
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departments and the government as a whole. First, if departments cannot integrate this information, they cannot 
give Parliament and the Canadian public a realistic picture of how much it has actually cost to produce a given 
result or deliver a given level of service. Second, without integrated information, senior management cannot 
readily forecast the downstream costs of major decisions — for example, a decision to increase or decrease the 
level of service of a program. 


13.8 We also found that departments had not established a clear vision or strategy for the transformations that 
are to take place — an essential first step in managing the changes. Similarly, departments have not determined 
the skills and capacities that are needed to carry out these changes or whether there is a gap between what is 
needed and what they currently possess. We also noted that departments had not implemented specific measures 
against which progress in developing strong financial management capabilities can be tracked. 


The response of the Treasury Board Secretariat, on behalf of the government and the five departments 
audited, is included at the end of this chapter. The response indicates that the recommendations are 
consistent with the current improvement plans and government expectations of sound management 
practices. 


The responses of Canadian International Development Agency, Environment Canada, Fisheries and Oceans, 
Health Canada and Transport Canada are included in the case studies in the chapter. The responses indicate 
an intent to address the gaps identified and that in many cases initiatives are already under way. 
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Canadian International 
Development Agency 


Managing Contracts and 
Contribution Agreements 


Chapter 14 - Main Points 


14.1 In CIDA’s Geographic programs, contracts and contribution agreements with Canadian executing 
agencies are used to implement development projects. Where CIDA used a competitive process for selecting 
executing agencies, with some exceptions the process was properly conducted. However, we observed instances 
where contracts did not comply with the Treasury Board Contracting Policy or the Government Contracts 
Regulations. 


14.2 An authority framework, similar to that for the contracting process, is not in place for CIDA’s 
contribution agreements. The terms and conditions for grants and contributions related to the Geographic 
programs are very general and provide no direction on how and when to use contribution agreements. They 
include a provision that contributions are to be approved in accordance with regular departmental procedures and 
authorities; however, exceptions can be dealt with internally by CIDA. Consequently, CIDA’s use of contribution 
agreements to select executing agencies often varied from its stated internal policies or practices. CIDA can select 
executing agencies by means of contribution agreements, which are, in effect, the same as sole-source contracts 
that would not be permitted under Government Contracts Regulations. This was the case in about half of the 
contribution agreements we examined. 


14.3 CIDA project officers attach considerable importance to monitoring the agreements under their 
responsibility. They commonly use monitors under contract to review and report on progress, and they insist on 
receiving reports from Canadian executing agencies as required. 


14.4 For the Voluntary Sector program, the Canadian Partnership Branch obtains reasonable information on 
the financial health of its Canada-based partners. However, limited information is received on projects that were 
funded, on the amounts spent on them, and on results obtained. CIDA bases its funding primarily on historical 
levels rather than on partners’ performance. More meaningful and accurate information on the Canadian 
Partnership Branch is needed in CIDA’s Performance Report to Parliament. 


Background and other observations 


14.5 CIDA is responsible for managing about $1.8 billion of Canada’s international assistance. Approximately 
$700 million of that amount goes to Geographic Branches for programs aimed at countries in Asia, Africa and 
Latin America. Most of this amount is spent through contracts and contribution agreements with third parties, 
referred to as executing agencies, to deliver development assistance projects. Another $260 million goes to the 
Canadian Partnership Branch for grants and contributions to organizations to carry out their own aid programs. 


14.6 In 1998, our Office published a report that commented on results-based management related to 
Geographic programs. We concluded that progress in managing for results was evident but uneven. 


14.7 This audit focussed on how CIDA’s Geographic Branches manage contracting and other types of 
agreements for goods and services, including the selection of Canadian executing agencies to deliver projects. It 
also examined the control framework for agreements in the Canadian Partnership Branch’s Voluntary Sector 
program. The audit aimed to assess whether CIDA’s contracting/contribution agreement processes respect 
Government Contracts Regulations, Treasury Board guidelines and its own policies; whether they are fair and 
transparent; and whether they meet operational requirements and development needs. 
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The Agency’s responses to our recommendations are included in this chapter. The Agency accepts the 
recommendations and indicates the actions that it is taking or intends to take to address them. 
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Health Canada 


First Nations Health: Follow-up 


Chapter 15 — Main Points 


15.1 Health Canada has initiated action to address our observations and recommendations on First Nations 
Health (1997 Report, Chapter 13). However, we are concerned that the Department has not yet made sufficient 
progress to fix many of the problems we identified. Continued and sustained effort is required to ensure that all the 
recommendations are fully implemented. 


15.2 We found that: 


e the point-of-service system, a key mechanism in the administration of pharmacy benefits and the 
control of prescription drug use, is now fully operational. It provides drug utilization warning 
messages to assist pharmacists in deciding whether to dispense prescriptions. As a result, a number of 
prescriptions have not been filled; and 


e apre-determination process for dental benefits, implemented nationally since 1997, has resulted in 
substantial savings. 


[k5y3! However, we observed that: 


e the management of contribution agreements for the delivery of community health programs is still 
weak; 


e the management of transfer agreements still needs improvement; and 


e Health Canada has not adequately monitored the drug utilization warning messages that pharmacists 
have overridden. Nor has it undertaken sufficient analysis of the effectiveness of the messages to 
determine whether any adjustments are warranted. 


Background and other observations 


15.4 Health Canada delivers health services and other related services to over 600 First Nations communities. 
It delivers them mainly through community health programs (including those transferred to community control) 
and the Non-Insured Health Benefits program, which covers such benefits as prescription drugs, dental care, 


medical transportation, and other benefits. Together, these services represented spending of just over $1.0 billion 
in 1998-99, 


i335 Our 1997 audit found that Non-Insured Health Benefits program expenditures were not well managed 
and, in most areas, not properly controlled. In particular, significant weaknesses in the management of pharmacy 
benefits allowed clients to access extremely high levels of prescription drugs. We noted numerous reports of 
prescription drug addiction and prescription-drug-related deaths of First Nations individuals in several provinces. 
We also found weaknesses in the management of contribution agreements for the delivery of community health 
programs as well as in the management of transfer agreements. 


15.6 Our follow-up found that the Department has made limited progress in ensuring that reports required 
under separate contribution agreements are provided. We also found that the Department’s performance reports 
contain limited information on the status of community health programs, including outcomes achieved. 


17 With respect to transfer agreements, we found that community health plans are still not being updated 
When agreements are renewed, and requirements for audits, annual reports and evaluations often are not met. 
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15.8 The Department developed and implemented a protocol for following up with clients, physicians, 
pharmacists and professional bodies on cases that suggest possible prescription drug misuse. This has had some 
positive impact, with some regions starting to show a decline in the number of cases of individuals accessing large 
amounts of prescription drugs. However, as management was unsure of the appropriateness of the approach in the 
absence of either client consent or specific statutory authority for the Non-Insured Health Benefits program, this 
intervention was stopped in May 1999. There are still cases of program clients accessing large amounts of 
prescription drugs, and these require follow-up. 


15.9 The Department was slow to develop and incorporate an audit strategy based on an appropriate 
assessment of risks in its new pharmacy and dental claims-processing contract. It also failed to ensure that the 
contractor perform the on-site audits of pharmacies and dental providers that the contract requires. We found few 
such audits undertaken since December 1998 to provide assurance that expenditures claimed had been incurred for 
the intended purposes. 


15.10 Progress to improve efficiencies in administering medical transportation benefits has been limited. 


Health Canada’s responses to our recommendations are included in this chapter. The Department is 
continuing with corrective action in response to our 1997 recommendations and has agreed to take action in 
response to our two new recommendations. 
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Follow-up of Recommendations 
in Previous Reports 


Chapter 16 — Main Points 


16.1 This chapter presents our follow-up of six audits originally published between 1995 and 1998. In the final 
volume of this year’s Report, due to be tabled in December 2000, we will publish an additional follow-up chapter 
and provide a general conclusion of the government’s progress in addressing the concerns we raised in previous 
reports. 


16.2 The policy and legislative environment of the financial services sector has changed considerably since 
our 1995 and 1997 Reports. Canada’s regulatory system for this sector is sound and the Office of the 
Superintendent of Financial Institutions (OSFI) has taken many satisfactory steps to deal with our previous 
recommendations. However, with all the expected changes that will affect the financial services sector, OSFI’s 
ability to supervise and regulate could be challenged in the short term. Furthermore, in this changing environment, 
recruiting and retaining employees will become a bigger challenge. 


16.3 Although progress has been slow on addressing the recommendations of our November 1995 chapter — 
Revolving Funds in the Parliamentary System: Financial Management, Accountability and Audit — recent 
initiatives by the Treasury Board Secretariat substantially address our recommendations. 


16.4 The Canada Industrial Relations Board, formerly known as the Canada Labour Relations Board, has 
made progress in addressing our concerns about financial control problems. Payment to former members was the 
single biggest expenditure of the Canada Industrial Relations Board’s $1.7 million in transition costs. We also 
observed that the government, as a whole, has not taken adequate action to improve the accountability framework 
governing travel and hospitality expenditures of senior Governor in Council appointees. 


16.5 National Defence has made significant progress in addressing its lack of plans and priorities for its capital 
acquisition program. The management of individual major capital projects has also improved. The Department has 
gone beyond its original commitments to make improvements. Nevertheless, officials estimate that the 

Department is still about $750 million a year short of the amount needed to modernize and maintain readiness. 
The “hard choices” referred to in 1998 have been identified, but not all of them have been made. 
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Other Audit Observations 


Chapter 17 - Main Points 


17.1 The Auditor General Act requires the Auditor General to include in his Reports matters of significance 
that, in his opinion, should be brought to the attention of the House of Commons. 


17.2 The “Other Audit Observations” chapter fulfils a special role in the Reports. Other chapters normally 
describe the findings of the comprehensive audits we perform in particular departments, or they report on audits 
and studies of issues that relate to operations of the government as a whole. This chapter reports on specific 
matters that have come to our attention during our financial and compliance audits of the Public Accounts of 
Canada, Crown corporations and other entities, or during our value-for-money audits. 


6 A The chapter normally contains observations concerning departmental expenditures and/or revenues. The 
issues addressed generally involve failure to comply with authorities, and the expenditure of money without due 
regard to economy. 


17.4 Observations reported in this chapter cover the following: 


e Space was leased at an excessive cost for a Canada Business Service Centre in Sydney, Nova Scotia 
that was never operated as intended; 


e Inappropriate netting of benefit payments obscures the true size of government revenues and 
expenditures and complicates the evaluation of fiscal measures; 


e Government programs and spending for Parc Downsview Park Inc. lack clear parliamentary 
authority; 


e Non-recovery of expenditures for safe drinking water on Indian reserves affected by Manitoba Hydro 
development; and 


e Significant risk that a $113 million relocation project will not adequately address the needs of the 
Innu. 


Li Although the individual audit observations report matters of significance, they should not be used as a 
basis for drawing conclusions about matters we did not examine. 
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Governance of Crown Corporations 


Chapter 18 - Main Points 


18.1 The management of Crown corporations has improved since the Financial Administration Act was 
amended in 1984 to strengthen their control and accountability framework. Nevertheless, further improvement is 
needed in some important areas like strategic and corporate planning and the measurement and reporting of 
performance. We found a need for special attention in three areas that are central to the way Crown corporations 
are governed: 


e Boards of directors of Crown corporations need to be strengthened. They reflect Canada’s diversity 
but lack other key skills and capabilities that are needed to function effectively and to carry out their 
important responsibilities under the Financial Administration Act for the affairs of the corporation. 
Corporations need to better define their requirements for skills and capabilities and communicate 
them to the government; the government needs to act on those requirements. Boards of directors also 
need to be more engaged in the selection of their chair as well as the corporation’s chief executive 
officer (CEO). Without meaningful board involvement in the selection of the chief executive officer, 
his or her accountability to the board is weakened and corporate governance as a whole suffers. 


e Audit committees in Crown corporations play a crucial role in financial reporting, risk management, 
and internal control. They are the “engine” of the board of directors. Yet half of the audit committees 
we examined were operating below an effective level. Serious weaknesses in an audit committee can 
undermine the overall strength of the board. 


e The government has a limited capacity for reviewing and challenging Crown corporation corporate 
plans as a basis for approving them. Corporate plans set out the strategic direction of a Crown 
corporation and are intended to be the cornerstone of the Crown corporation control and 
accountability framework under the Financial Administration Act. The government needs to 
strengthen its capacity to review and challenge these plans since, once approved, they are the basis 
for holding Crown corporations accountable for conforming to government policy and for their 
overall performance. Furthermore, there is a need for a more systematic and regular review of the 
relevance of Crown corporation mandates. 


18.2 Weaknesses in all of these areas impede the successful implementation of Part X of the Financial 
Administration Act and the quality of Crown corporation governance. They have been raised many times before, 
in Auditor General reports and other external studies and reports, but the weaknesses remain. Crown corporation 
CEOs and chairs view the resolution of these issues as critical and they need to be addressed with urgency. Other 
industrialized countries have moved ahead of Canada in these areas of Crown corporation governance. 


18.3 Corporate governance refers to the process and structure for overseeing the direction and management of 
a corporation so that it carries out its mandate and objectives effectively. It is critical that a Crown corporation, as 
a public sector body, be governed well if taxpayers’ money is to be well spent. Good corporate governance is 
important to maximize performance, prevent financial losses and help to achieve the corporate mandate. But it is 
in times of difficulty, turbulence and change that good governance is most critical. 


Background and other observations 


18.4 Crown corporations are distinct legal entities wholly owned by the government. They operate in many 
sectors, including transportation, agriculture and culture. They have more autonomy to manage than most other 
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government entities so they can operate in a more commercial manner. A board of directors oversees the 
management of each corporation and holds management responsible for the corporation’s performance. The 
government retains power and influence over Crown corporations in areas like appointments, remuneration for 
chief executive officers and directors, and approval of plans and budgets. 


18.5 Crown corporations account for a significant portion of government activity. There are currently 

41 federal Crown corporations (not including subsidiaries), employing 70,000 people. In aggregate (excluding the 
Bank of Canada), they manage $68 billion in assets and $61 billion in liabilities. While Crown corporations 
represent a significant opportunity to achieve public policy and other goals and to generate revenue, they also 
represent significant exposure to potential financial losses and other risks. 


18.6 In addition to our continuing work as auditors and examiners of most Crown corporations, we looked in 
detail at a representative group of 15 Crown corporations. We also compared Canadian Crown corporations with 
state-owned enterprises elsewhere, and used the results of other research, studies, audits and roundtables to 
support our findings. In addition, we summarized the results of special examinations — a type of value-for-money 
audit of Crown corporations — carried out between 1984 and 2000. 


In view of the fact that many of the recommendations deal with Governor in Council discretion, no detailed 
responses to the recommendations were made. Overall, the Privy Council Office states that the government 
has made strides in a number of corporate governance areas, and is committed to examining measures to 
address areas needing improvement and to ultimately strengthening governance in Crown corporations. The 
Privy Council Office also states that it is paramount to ensure that appointments to Crown corporations 
result in strong boards of directors, and that the appointment process for Crown corporations be inclusive of 
all interested parties, bearing in mind that appointments remain at the discretion of the government. 


Reaction from Crown corporation chairs and CEOs indicate that, for the most part, they agree with the 
recommendations directed specifically at them. 
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Reporting Performance to Parliament 


Progress Too Slow 


Chapter 19 - Main Points 


19.1 Federal departments and agencies have made some progress in reporting on their performance to 
Parliament, but we are disappointed at its present pace. Continued at this rate, it would take too long for good 
reporting to become routine. We recognize that reporting on performance is not easy, and it takes time. Still, we 
believe that the effort deserves more attention: members of Parliament need information on what Canadians are 
getting for their taxes in order to hold the government to account. Good performance reporting has become an 
essential part of modern democratic systems. 


19.2 To that end, it may be time for legislation. Several jurisdictions in Canada now have legislation requiring 
government to report on its performance to the legislature. Legislation highlights the importance of good reporting 
and puts it on a more permanent and stable footing. 


19.3 Our examination of the Estimates documents — reports on plans and priorities and departmental 
performance reports — of 47 departments and agencies over the past four to five years found the following: 


e While there has been progress, few departments set out concrete statements of what they expect to 
achieve and then report back specifically against those expectations. 


e In reporting what they have accomplished, departments focus too much on listing activities and 
outputs and too little on linking them to and reporting on the intended outcomes. 


e Reporting makes too little use of evaluation findings on the results that activities are accomplishing 
and on how programs are contributing to outcomes. 


e There is little linking of financial and non-financial performance information; most accomplishments 
are not costed. 


e Performance reports lack balance. Most departments report only the “good news” and make little 
mention of performance that did not meet expectations. As members of Parliament have said 
repeatedly, reports that present only good news are not credible. 


19.4 We noted that the Treasury Board Secretariat has provided leadership for reporting. And we did find 
pockets of good reporting and a number of strengths in the present regime of reporting. Nevertheless, after five 
years of experimenting, departmental reporting overall has only improved modestly. 


19.5 Three factors stand out as contributing to the current weak state of reporting: 
e basic principles of good reporting are not understood or applied; 
e performance reporting has political dimensions; and 
e — there are few incentives for good reporting or sanctions for poor reporting. 


19.6 In addition to considering legislation, in our view the government needs to make the following changes 
in the present reporting regime: 


e reach agreement on principles for reporting performance information; 
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e give more attention to the right incentives for incorporating evidence-based approaches into 
reporting; 


e ensure stronger oversight by the Treasury Board Secretariat; and 


e seek to enhance the involvement of parliamentarians in reviewing performance plans and reports. 


Background and other observations 


19.7 Federal departments submit performance reports to Parliament in the fall every year, outlining what they 
have accomplished. The reports are subject to consideration by their respective parliamentary standing 
committees. Review of the reports at that time would enable parliamentary committees to make recommendations 
for the upcoming spending decisions to be announced by the government later, in the Budget. In the spring, after 
the Budget, departments submit reports on their plans and priorities, describing what they are setting out to 
achieve in the upcoming year and beyond. 


19.8 In 1997 we first reported on this new regime for reporting to Parliament. We found that a good start had 
been made and that the basic framework used for reporting was sound. 


19.9 Credible reporting on performance has a number of benefits: it can enhance accountability and trust in 
government, provide information for improved programs and services and better parliamentary scrutiny of them, 
and serve as an incentive for departments and agencies to manage for the results they have set out to achieve. 


19.10 On two occasions in recent years, the Standing Committee on Procedure and House Affairs has issued 
reports with recommendations on ways to provide for greater involvement of parliamentarians in the review of the 
Estimates. 


19.11 The President of the Treasury Board annually tables, along with the departmental performance reports, a 
report in Parliament on government performance measurement, reporting and results-based management. In recent 
years it has provided useful performance information on collective results — areas where several departments or 
several governments are working collectively toward a result. 


19.12 The Treasury Board Secretariat has taken important steps toward more electronic reporting. It needs to 
encourage departments to make greater use of this vehicle, and gather and share good practices. 


The response of the Treasury Board Secretariat, on behalf of the government and the departments audited, 
is included at the end of this chapter. The Secretariat has responded positively to our chapter and 
recommendations therein. The Secretariat accepts our recommendations and indicates the actions that, 
where appropriate, it is taking or intends to take to address them. 
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Managing Departments for 
Results and Managing Horizontal 
Issues for Results 


Chapter 20 — Main Points 


20.1 In our examination of five large federal departments and 10 programs, we found that several departments 
have made tangible efforts to focus on results. There was evidence of commitment and support by senior 
management, extensive planning activities, identification of key performance indicators and some measuring 
against them. 


20.2 However, we found that only 4 of the 10 programs have information on results that is used routinely. In 
most departments we examined, the persistent state of planning for results — rather than actually managing for 
results — has meant that progress in using information on results has been limited since we reported on this issue 
in 1997. It is time for departments to move beyond planning. There is a need for more widespread measurement of 
the results they accomplish and use of that information to improve programs. 


20.3. The government needs to ensure that the evaluation function plays a more central role in its move toward 
managing for results. The evaluation capacity in many departments has atrophied at a time when more effort, not 
less, is needed to measure the results of programs. 


20.4 More and more, the government needs to manage initiatives that span two or more federal departments. 
“Horizontal” issues like the Federal Disability Agenda are often key priorities of government. Our study of three 
horizontal issues found that, like many departmental programs we audited, they were not being managed for 
results. 


20.5 Over the past several years, the government has been clear about the importance of making results central 
to good management and accountability, and this vision is increasingly being accepted across government. 
However, bringing that recognition into practice will require central agencies to play a stronger role. They need to 
provide encouragement and support for departments to move beyond merely planning for results. We found the 
Treasury Board Secretariat’s leadership too limited and its support too dispersed to be of real help to departments 
trying to manage for results. Nor does the Secretariat have a strategy for dealing with horizontal issues. It needs a 
strategic approach to these issues, given their growing presence in government. 


Background and other observations 


20.6 Managing for results allows managers to make changes once they know what is working and what is not. 
It represents a significant difference in the way government programs are managed. Managers can then pay more 


attention to finding out whether programs are meeting their objectives and less to only carrying out activities or 
setting up structures and processes. 


20.7 The concept of managing for results is now widely accepted across government. As a first step in putting 
it into practice, most departments had to realign their management structures and processes according to the 
results they were seeking, and not to their programs or activities. Each department had to establish 
department-wide objectives and think through how it wanted to structure itself in terms of the results it expected. 


Most of the departments we audited were engaged in significant efforts to plan how they would manage for 
results. 


20.8 In managing a horizontal issue for results, it is critical to have a co-ordinating function that is supported 
by senior management (particularly in a lead department) and that has enough resources to do the job. 
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20.9 Departments receive limited practical guidance on managing for results, and there is little sharing of 
experience. They need more encouragement to move beyond planning for results; there is also a need to identify 
champions. 


20.10 While managing for results presents challenges, there are known, practical ways to deal with them. They 
require that people embrace the idea of learning from experience — which includes learning from mistakes. It 
involves bringing more rigour to understanding past experiences and then adjusting practices in light of this 
learning. 


The response of the Treasury Board Secretariat, on behalf of the government and the departments audited, 
is included at the end of this chapter. The Secretariat accepts our recommendations and indicates the actions 
that it is taking or intends to take to address them. 
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Post-Secondary Recruitment 
Program of the Federal Public Service 


Chapter 21 — Main Points 


21.1 Our review of the Post-Secondary Recruitment program in the federal public service found that the Public 
Service Commission of Canada is recruiting qualified candidates. But there are too few of them given the 
significant number of public service executives, professionals and managers who will be eligible to retire in the 
near future. The government has to immediately address the recruitment issue so that it can continue to deliver 
programs and provide services of quality to Canadians. 


21.2. We found weaknesses in the program’s efficiency and effectiveness in the following areas: 


e Forecasting of recruitment needs is not clearly spelled out. Without a clear idea of the number and 
type of employees needed, it is difficult to establish clear recruitment levels and develop appropriate 
recruitment strategies. 


e The recruitment levels under the program have not been met consistently. 


e In light of the increased competition for university graduates, the Commission needs to be a more 
aggressive recruiter of graduates. There has to be better promotion of the public service as a career 
choice, increased visibility on campus and improvement in scheduling and timeliness of recruitment. 


e Very few departments participate in the program, which limits its capacity to renew the public 
service. 


e The Commission does not establish and manage inventories of qualified candidates for departments. 


e The program and its activities have not been evaluated consistently and systematically to improve 
their effectiveness and efficiency. As a result, it makes it difficult for the government to take 
appropriate corrective action on recruitment. 


e Reporting of results is not adequate. 


21.3 We are concerned about the government’s ability to address the recruitment challenge. 


Background and other observations 


21.4 The public service has become significantly older. Seventy percent of executives could retire by 2008, 
and the feeder groups, the people who could eventually replace them, have a similar retirement profile. Public 
servants under 35 years of age are underrepresented. 


21.5 For existing senior executives, it took an average of 10 years for them to move from a professional entry 


level to an executive level. Recruitment has, therefore, become imperative to ensure a well-functioning public 
service in the future. 


21.6 The public service has changed and requires a higher proportion of knowledge workers than in the past. 
Today, these workers constitute 55 percent of public servants. 


21.7 Recruitment has been an issue in the public service for at least 10 years. Many reports, and more recently 
the 1999 Speech from the Throne, have reported the need to recruit for the public service because of 
demographics and other factors, such as increased competition for qualified candidates. 
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21.8 It is recognized that an effective and non-partisan public service is essential to the strong functioning of a 
democratic country. To remain a vibrant institution, the public service must be able to recruit and retain 
well-qualified employees. 


21.9 The government recognizes that recruitment in the public service is a priority. In his March 2000 annual 
report, the Clerk of the Privy Council stated the following: “We need to start recruiting people now so that we 
have the time to groom and nurture skills and so new recruits will have time to benefit from the depth of 
experience and knowledge of public servants across the country.” 


21.10 We are concerned about the government’s ability to address the urgent challenge of recruitment for the 
years ahead. 


The responses of the Treasury Board Secretariat, on behalf of the government and the departments audited, 
and of the Public Service Commission are included at the end of this chapter. The responses recognize the 
importance of recruitment and indicate plans to address the issues identified. 
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Development of the Universal 
Classification Standard 


A Follow-up 


Chapter 22 - Main Points 


22.1 The Treasury Board Secretariat has made major progress in implementing our 1996 audit 
recommendation on the development of the Universal Classification Standard for the federal public service. This 
project is fundamental to the reform of human resource management, particularly for staffing and compensation. It 
is a monumental and highly complex undertaking that requires much time. The present system is outdated and no 
longer meets the requirements of today’s public service and tomorrow’s. 


22.2 In the last few years, the Secretariat has designed a standard intended to measure the content of diverse 
types of work in the public service, and has subjected it to a series of tests in departments. Since 1998, 
departments have been writing new work descriptions for all positions and evaluating them against the Standard. 
In the course of this project, the Treasury Board Secretariat has developed and issued guides and methodologies 
based on best practices in job evaluation. 


22.3 Despite these achievements, the government still has to address significant issues: 


e The Treasury Board Secretariat needs to ensure that the new Standard is being applied appropriately 
before fully converting to it. 


e Departments need to ensure that their work descriptions are accurate and their position evaluation 
results valid and adequate. 


e The Secretariat needs to perform further testing, based on accurate and up-to-date evaluation data, to 
establish the extent to which the Standard meets its stated objectives to be universal and 
gender-neutral. 


e The government needs to ensure that reasonable and realistic financial resources are available to 
convert to the new Standard. 


Background and other observations 


22.4 — A classification standard is a measuring tool for determining the relative value of positions in an 
organization. The Universal Classification Standard, now being developed, must be capable of evaluating all 
positions in the public service on the basis of work content. The results will be used to establish a hierarchy of 
positions and the compensation that reflects their requirements. 


22.5 The Standard will apply to all Public Service employees for whom the Treasury Board is the employer, 
with the exception of the executive group. At August 2000, there were some 150,000 positions in 72 separate 
occupational groups, in some 65 departments and agencies subject to the new classification system. The 


development of a new classification standard has to meet the requirements of the Canadian Human Rights Act on 
equal pay for work of equal value. 


22.6 The existing classification standards do not reflect today’s emphasis on serving the public and managing 
for results. They also do not make reference to the fact that public servants work more and more in partnership 
with other levels of government and the private sector. 
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22.7. ~The development of a new classification standard was launched in 1991 as part of the Public Service 
2000 initiative. In 1996, we reported serious deficiencies (May 1996 Report, Chapter 5). Pursuant to our 
recommendations and the findings of its own review, the government began to improve the Universal 
Classification Standard. This follow-up chapter is a progress report on the initiative to date. 


The Treasury Board Secretariat has responded positively to our findings and recommendations and has 
affirmed that progress will continue to meet the Universal Classification Standard objectives. 


a 
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Information Technology 


Acquisition of Goods and Services 


Chapter 23 - Main Points 


23.1 In auditing acquisition activities (procurement) for large information technology projects in the 
government, we found that two recent acquisitions followed some essential elements of a procurement approach 
and project management framework designed to improve large technology projects. The approach emphasizes the 
benefits to be derived from a project, rather than focussing on a specific solution to a business problem. Although 
both projects are still in the construction phase, they have progressed and delivered as planned. While full 
implementation and deployment of the new systems are not assured, the results to date are encouraging. 


23.2 We found, however, that the acquisition activities still spanned 18 months or more. The government 
needs to improve the timeliness of those activities to adequately support its objective of putting government 
on-line by 2004. 


23.3 We also examined the acquisition process for the routine supply of microcomputers, equipment for local 
area networks and software. We found that the government has yet to finish replacing its discontinued method of 
supply for routine, low-volume purchases of software licences and software solution services. As a result, user 
departments and agencies do not have a ready, prenegotiated means for low-value purchases of certain software 
products and services. 


23.4 We also found that the replacement method as developed thus far is inadequate, and there is still a risk 
that the government will continue to acquire installed bases of software on a piecemeal and non-competitive basis. 
Where user departments and agencies have done so, the government can become vulnerable to the filing of 
complaints with the Canadian International Trade Tribunal in relation to contracts that extend the use of the 
software to the entire department or involve department-wide maintenance and upgrade of the software. 


23.5 Acquiring software licences and software solution services appropriately continues to be a challenge. The 
government needs to consider a strategy for acquiring software as a technology investment. 


Background and other observations 


23.6 Our data analysis showed that the significant expenditures involved in the acquisition of information 
technology (IT) goods and services have been on the rise. We estimated that in 1998-99, spending on IT 
amounted to about $3.4 billion, including salary costs. 


Zaet The method of supply for routine, low-volume purchases of microcomputers and products related to local 
area networks has merit and adds value to the acquisition process. Nevertheless, in our view there are 
opportunities for cost savings of about $10 million a year by taking more advantage of consolidated purchases. 


23.8 Our audit looked at eight user departments and agencies. We conducted a test of their compliance with 


the procurement authority delegated to them. We found a few minor deviations but no indication of blatant or 
widespread misuse of delegated authority. 


23.9 Recent determinations by the Canadian International Trade Tribunal (CITT) were the main reason for 
changing the standing offer method of supply for routine, low-volume purchases of software licences and software 
solution services. Moreover, even though CITT cases have been few, government users and the IT community 
have indicated that they have generated intense scrutiny of all technology acquisitions in government in order to 
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avoid potential complaints to the Tribunal. In our view, there is a need to better respect the government’s 
contracting objectives — meeting operational requirements with due regard to value for money while also 
respecting the principle of open and fair competition by complying with government regulations and trade 
agreements. 


The government is generally supportive of the recommendations contained in the chapter. 
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Federal Health and Safety 
Regulatory Programs 


Chapter 24 - Main Points 


24.1 The overall objective of health and safety regulatory programs is to proactively protect Canadians from 
risks to health and safety — to catch the problem before it happens, and if it happens, to minimize the 
consequences. 


24.2 The Canadian Food Inspection Agency, Health Canada, Environment Canada, Transport Canada, the 
Canadian Nuclear Safety Commission and the National Energy Board administer major federal health and safety 
regulatory programs. We estimate that in 1999-2000 the regulatory programs administered by these organizations 
spent about $1.2 billion and employed some 12,000 people. They administer about 85 Acts and 250 regulations. 
Their objectives range from controlling toxic chemicals to maintaining the safety of food, drugs and nuclear 
power plants. 


24.3 The objective of the government’s regulatory policy is to promote the design and implementation of 
effective regulatory programs. Because performance measurement is weak, there is insufficient information to 
assess the cost effectiveness of health and safety regulatory programs. 


24.4 Over the past decade, our audits of federal health and safety regulatory programs have found many 
instances where the regulatory authorities have not met the expectations of the government’s regulatory policy. 
While effectiveness cannot be judged solely on the basis of adherence or lack of adherence to the policy, well 
structured programs increase cost effectiveness and reduce the risk of regulatory failure. 


24.5 Improving the structure and implementation of federal health and safety programs will require action 
government-wide and by the responsible regulatory authorities. In particular, we emphasize the need to ensure the 
following: 


e the government explains to Canadians and the federal regulatory and inspection community the 
priorities for health and safety regulatory programs, particularly the balance the government has 
reached between the objective of protecting Canadians, addressing budget concerns and meeting 
economic objectives; 


e reliable information is available on the level of risk and the extent to which it is and can be 
controlled; 


e based on priorities determined by risk assessment, regulatory authorities have sufficient financial and 
human resources; 


e the government identifies major health and safety objectives that can only be achieved through 


interdepartmental co-operation, and authorities assess officials on the achievement of these 
objectives; and 


e the government annually submits reports to Parliament on the overall effectiveness of health and 
safety regulatory programs, including reports by lead regulatory authorities on the achievement of 
objectives that require significant interdepartmental co-operation. 


46 Report of the Auditor General of Canada — December 2000 


Background and other observations 
24.6 We undertook this audit to identify the following: 
e major trends and challenges faced by health and safety regulatory authorities; 
e patterns of recurring strengths and weaknesses in structure and implementation; and 


e measures that could be taken to make significant improvements in the structure and implementation 
of regulatory programs. 


24.7 We used key elements of the government’s regulatory policy as a basis for organizing information on 
patterns of strengths and weaknesses. Our findings are based on the results of our current audits of food inspection 
programs, nuclear power plant regulation, the regulatory regime for biologics of Health Canada, and follow-up 
audits of animal health and plant protection, safety regulation for the air navigation system of Transport Canada, 
and onshore pipeline regulation. They are also based on audit findings from previous Reports of the Auditor 
General and the Commissioner of the Environment and Sustainable Development. As well, we have taken into 
account the findings of parliamentary committees, government reports and reports of non-government 
organizations. 


24.8 Canadians are concerned about health and safety risks. Crises or regulatory failures heighten these 
concerns. However, the government cannot eliminate all risks. Canadians need to be provided with understandable 
information on health and safety risks and consulted on the choices to be made. 


24.9 The public’s confidence in the government’s use of science and technology to protect the health and 
safety of Canadians has been shaken by recent crises, for example, the concerns about the safety of the blood 
supply reviewed by the Commission of Inquiry on the Blood System in Canada (Krever Commission). The use of 
independent expert advisory committees could be expanded to provide advice and enhance the credibility of the 
regulatory effort. 


24.10 Under the Constitution and in practice, the federal government often shares responsibility for the 
protection of health and safety with the provinces. Co-operation is fast becoming mandatory, rather than a matter 
of choice. Increasingly, risks are of a global nature, and multinational action is required. Standards and regulatory 
approaches among countries are being harmonized through trade treaties or international agreements. This trend 
means that Canadian regulatory authorities need to co-operate to effectively present a Canadian position in 
international exercises for setting standards. For these reasons, it is in the interest of all parties to work together 
and to participate in the development of a national health and safety regulatory plan. 


The Privy Council Office states that while the government believes its regulatory policy is sound, it shares 
our Office’s concern. It recognizes the need to ensure that regulatory authorities have the capacity to meet 
the expectations of the policy — to properly develop and to appropriately implement regulations and 
regulatory programs. 
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Canadian Food Inspection Agency 


Food Inspection Programs 


Chapter 25 - Main Points 


25.1 The Agency’s food inspection programs are reviewed and generally well regarded by most foreign 
countries that import our products. This feedback provides a degree of assurance that the Agency’s food inspection 
programs are contributing to the safety of the Canadian food supply. 


25.2 Since its creation in April 1997, the Agency has undertaken a number of initiatives to improve the 
efficiency and effectiveness of federal food inspection — a major reason for its creation. These initiatives include 
the development of an Integrated Inspection System (IIS) to promote greater consistency between inspection 
programs; and the introduction of an approach based on the principles of hazard analysis and critical control 
points (HACCP). The HACCP-based approach is a tool to improve the safety of food through the assessment of 
hazards and the establishment of control systems that focus on prevention, rather than mainly the reliance on 
end-product testing. 


25.3 While progress has been varied and in a few cases faltered, the Agency has made progress in many of the 
initiatives. We noted that the Agency has had difficulty in establishing a process to support risk-based resourcing. 
As a result, the Agency cannot demonstrate that it has appropriately resourced its food inspection programs based 
on risk. Such a process is needed particularly in the imported food sector, which is worth $15 billion and is 
growing, and the non-federally registered sector, which represents almost one-half of the food-processing industry. 
We also noted in the non-federally registered sector that the Agency has changed its inspection approach without 
adequately assessing overall risks and the options available for managing the sector. Also of concern is the lack of 
a formal strategy for the implementation of the HACCP-based approach and the setbacks encountered in the IIS. 


25.4 We identified problems with the Agency’s compliance activities. We examined 21 inspection files from 
establishments that had issued food recalls or had been prosecuted in the last two years. In 1999-2000, the Agency 
participated in 243 recalls and had 59 successful prosecutions. We found that in 16 cases, the same or similar 
problem persisted for many months and in some cases, years. The compliance actions taken were not sufficient to 
achieve the Agency’s goal of timely correction of the compliance problem either because of the limitations in the 
legislation or a failure by the inspector to take more serious compliance action. 


25.5 Measuring the success of initiatives and more broadly the performance of the Agency in achieving its 
goals is also a concern. We noted that the Agency had not set expectations or measured performance to determine 
if its initiatives and activities have contributed to a more efficient and effective food inspection system. Our 
annual assessment of the performance information in the Agency’s annual report has consistently noted that 


readers are not provided with the information necessary to understand the extent to which the Agency is achieving 
its objectives. 


Background and other observations 


25.6 The Canadian Food Inspection Agency is the result of the amalgamation of food safety and inspection 
programs from three federal departments : Agriculture and Agri-Food Canada, Health Canada, and Fisheries and 
Oceans. It now has some 4,400 employees across the country and manages expenditures of $416 million. The 


Agency’s main activities focus on inspecting the food supply, but it also conducts activities related to animal 
health and plant protection. 
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25.7 The Agency is not solely responsible for food safety. It shares this responsibility with other federal 
departments and provincial, territorial and municipal authorities. Industry and consumers also play an important 
role. Health Canada is responsible for establishing policies and standards relating to the safety and nutritional 
quality of food. The Agency is responsible for delivering federal inspection programs that enforce these policies 
and standards. 


25.8 The Agency operates in a complex environment created by a number of factors, including changes in 
food risks, technology, inspection methodology, international requirements, consumer trends and consumer 
perception. 


25.9 Our audit focussed on assessing how the Agency manages food inspection to reduce risks to food safety. 


25.10 We found that a formal strategy for the implementation of the HACCP-based approach is needed. A plan 
for the transition period, including considering how to manage resources during the transition period needs to be 
developed. Further, insufficient information has been collected to allow the Agency to measure the success of 
implementing the HACCP-based approach in improving food safety. It could take many years to develop and 
implement the further redesign of the HACCP-based approach in the meat industry. This redesign would require 
industry to perform ante- and post-mortem detection with continuous government monitoring and oversight in the 
beef and pork industries. It could also take a long time to introduce a pathogen-reduction effort in these industries. 
This situation is in contrast with the important progress made in the poultry industry. 


25.11 The Agency is lacking important information on the incidence of food-borne illness in humans and the 
prevalence of pathogens in the food supply. This is complicated by the fact that the Agency is not responsible for 
gathering some of the information. However, without this information, it is more difficult to manage risks to food 
safety and measure the success of Agency initiatives and its contribution to the safety of the food supply. 


25.12 The Agency has consulted the provinces, industry and the public on some, but not all, significant 
initiatives and issues. As a result, the Agency has missed opportunities for broad public debate on important 
questions, including the following: How much inspection attention should be paid to the non-federally registered 
sector given its mandate? Should the Agency continue to provide industry with considerable flexibility in 
implementing the HACCP-based approach? Should it be leading internationally in implementing the further 
redesign of the HACCP-based approach in the meat industry? It is important that the Agency engage stakeholders, 
including Parliamentarians and the public, in a debate on these questions. 


The Canadian Food Inspection Agency’s responses to our recommendations are included in the chapter. The 
Agency accepts all but one of our recommendations and has indicated a number of actions it is taking or 
intends to take to address them. The Agency disagrees with our recommendation relating to the 
non-federally registered sector. Its position is reflected in its response following paragraph 25.82. 
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Health Canada 


Regulatory Regime of Biologics 


Chapter 26 - Main Points 


26.1 Our audit of Health Canada’s regulatory regime for biologics found that the Department is taking a 
reasonable approach in developing and implementing frameworks and approaches for regulating different 
biologics (blood and blood products, cells, tissues and organs of human or animal origin, vaccines and other 
biological drugs). Our work focusses on the management of the regime for regulating biologics. Therefore, we did 
not conclude on the effectiveness of specific regulations or on the safety of specific products. 


26.2 The Department faces important challenges. It needs to do the following: 


e ioestablish more formal guidance to determine the most appropriate regulatory approach for a given 
situation; 


e to maintain the currency of regulations; 
e to obtain sufficiently qualified staff to deal with the rapid technological advances in biologics; and 


e to implement sufficient databases to adequately process, analyze and disseminate information on 
adverse reactions and events. 


26.3 Overall, we found that the Department has adopted a proactive approach to identifying risks that could 
threaten the health and safety of Canadians in the area of biologics. Though the science for xenotransplantation 
(the use of live, non-human animal cells, tissues and organs in humans) is not perfected, the Department is being 
proactive in planning a regulatory regime for xenografts. However, many delays have characterized the 
implementation of the regulatory regime for transplantation of human tissues and organs. Transplants of these 
types have been performed in hospitals for a number of years. 


Background and other observations 


26.4 Health Canada is responsible for regulating biologics under the provisions of the Food and Drugs Act and 
Regulations. Accordingly, the Department’s objective is to ensure that biologics available to Canadians are safe, 
effective and of high quality. The Bureau of Biologics and Radiopharmaceuticals does much of this work with 
several other bureaus in the Therapeutic Products Directorate of the Health Products and Food Branch. As well, 
the Population and Public Health Branch conducts some surveillance activities. 


26.5 The safety of biologics is an important aspect of public health. Rapid advances in science have resulted in 
increasing regulatory challenges. Many of the federal laws that govern health and safety were developed decades 
ago. Today, science is yielding new products that up until recently were unimaginable. Some products and 
practices carry potential health risks and raise difficult questions that current legislation may not be fully equipped 
to address. It is important that regulations remain current. It is also necessary to ensure that the tools required to 


put in place regulations — or other interventions — are also current and sufficiently flexible to deal with changing 
demands. 


26.6 The Department follows a traditional approach with prescriptive regulations to regulate most biologics. 
This framework is well-established and offers some benefits. However, Health Canada believes that the 
framework is not sufficiently flexible to deal with emerging products and other technological advances in 
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biologics. Therefore, the Department is moving toward adopting a standards-based regulatory approach for blood, 
tissues and organs, and xenografts. Under this approach, third-party standards development organizations develop 
standards with Health Canada and other interested parties. 


26.7‘ The Processing and Distribution of Semen for Assisted Conception Regulations (the Semen Regulations) 
represented the only regulatory framework in biologics that incorporated by reference in the Regulations standards 
developed by a third party. While this approach appears to offer some benefits including more flexibility to deal 
with rapid changes in technology, there are some important issues that need to be addressed. The Department 
needs to ensure that compliance is verified and that accountability is assured for regulations referencing its own 
technical standards. The Department needs to apply lessons learned from the implementation of the Semen 
Regulations to future standards-based regulatory frameworks, as applicable. 


Health Canada’s responses to our recommendations are included in this chapter. The Department agrees 
with the recommendations and indicates the actions that it is taking or intends to take to address them. 
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Canadian Nuclear Safety 
Commission 


Power Reactor Regulation 


Chapter 27 — Main Points 


27.1 The Canadian Nuclear Safety Commission (CNSC) needs to improve its regulatory regime for power 
reactors to ensure that it continues to protect the health and safety of Canadians. 


27.2 While the CNSC continuously monitors the compliance of power reactor licensees with regulatory 
requirements, its regulatory activities are not based on a rigorous, well-documented system of risk analysis and the 
ratings it assigns for regulatory performance (acceptable, conditionally acceptable or unacceptable) are not clear. 
In addition, CNSC’s compliance and enforcement system is not yet complete. As a result, it cannot adequately 
demonstrate that it is achieving its safety objectives for the regulation of power reactors. 


pA Like other nuclear regulators, CNSC faces significant difficulties in recruiting and retaining qualified 
staff. Combined with its current regulatory regime, which relies heavily on the expertise and judgment of staff, the 
lack of human resource capacity could impact its ability to function adequately in the future. 


Background and other observations 


27.4 The Nuclear Safety and Control Act came into force on 31 May 2000. It created the Canadian Nuclear 
Safety Commission to replace the Atomic Energy Control Board. CNSC is responsible for regulating the use of 
nuclear energy in Canada to protect health, safety, security and the environment. Our audit focussed on the 
regulatory activities related to power reactor licensing and regulation. 


27.5 The CNSC has 440 staff. Its headquarters are in Ottawa. It maintains a site project office at each of the 
power reactor stations, where its staff monitor licensee compliance with regulations and the licence conditions. In 
fiscal year 2000, CNSC had total costs of $59 million. 


27.6 In July 1999, of some 440 power reactors in the world, 22 were in Canada — 20 in Ontario, one in 
Quebec and one in New Brunswick. There were 104 power reactors in the United States. 


The Canadian Nuclear Safety Commission agrees with our recommendations and is taking action to address 
them. 
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Follow-up of Previous 
Recommendations on Health and 
Safety Regulatory Programs 


Chapter 28 — Main Points 


28.1 This chapter presents our follow-up of three previously reported audits of federal health and safety 
regulatory programs. The Animal and Plant Health segment is a further follow-up to that reported in Chapter 28 of 
our 1998 Report. 


28.2 Since our 1996 audit on animal and plant health, a number of initiatives have been undertaken within the 
Canadian Food Inspection Agency to address our recommendations concerning the assessment of risk and 
alternative service delivery; however, work remains to be done. Because many of our recommendations were 
long-term in nature, we did not yet expect their implementation to be complete. Progress is satisfactory, except in 
several specific areas. 


28.3 Transport Canada has made satisfactory progress in the areas of risk analysis and audit and inspection. 
However, it has still not conducted or planned any formal review of the quality of Civil Aviation Daily Occurrence 
Reporting System (CADORS) data or any other safety performance data regularly supplied to it by NAV 
CANADA. The Department also has not yet established a formal process to reconcile the data it receives from 
NAV CANADA with the data from the Transportation Safety Board’s incident-reporting system to ensure 
completeness. 


28.4 The National Energy Board has made a good effort to meet its commitments and implement the 
recommendations. Progress has been slower than anticipated in some areas but is on track for full implementation 
in the near future. The Board has taken the tasks seriously and assigned senior staff to ensure successful 
completion. 
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Treasury Board Secretariat 


The Government's Annual 
Contracting Activity Report — 1998 


Chapter 29 — Main Points 


29.1 This chapter examines the Treasury Board Secretariat’s Annual Contracting Activity Report for the 
calendar year 1998. The Report provides Canadians with an annual statistical summary of the number and value 
of contracts let pursuant to the Government Contracts Regulations (GCR) and the government’s contracting 
policy. The value of the reported contracts in 1998 accounted for $13.4 billion (including NATO Flying Training 
in Canada Program of $2.99 billion) or more than one third of the $35.7 billion cost of running the government. 


29.2 The systems for gathering data for the Report capture most of what is required and have the potential to 
provide the base for a good and useful document. However, our audit found significant deficiencies in the Report 
with respect to validity, reliability, transparency and relevance. (We define these terms later.) 


29.3 Specifically, the deficiencies that we noted in the 1998 Report include the following: 


e Validity. Transactions relating to contract amendments, standing offers and credit card payments are 
included with information on primary contracting activities. The information on these transactions 
should be segregated in the Report. 


e Reliability. The audit found that information on the number and dollar value of contracts was 
incomplete. For example, the Report did not include at least $320 million worth of contracts that 
should have been reported. We also found that information on the classification of contracts was 
unreliable. 


e ‘Transparency. The 1998 Report does not provide enough explanatory information or links to it to 
enable the unfamiliar reader to understand how the Report was prepared and the meaning of key 
terms used in presenting the data. 


e Relevance. As was the case for previous reports, the 1998 Report was not published until more than 
a year after the end of the reporting period, thereby making its relevance questionable. In addition, it 
does not contain enough information in particular areas to meet the needs of the public and 
parliamentarians. These needs have been identified by parliamentary committees. 


29.4 The available data do not permit us to quantify the effect of several of these deficiencies on the Report; 
nor do the data permit us to assess the interaction among them. Accordingly, we cannot assess the extent to which 
these deficiencies affect the overall fairness of presentation of the Report. 


Background and other observations 


29.5 The need for fairly presented and relevant statistical information on the government’s contracting 
activities has been a long-standing concern of the Office of the Auditor General. We first raised the issue in 1982 
and more recently in reports in 1991 and 1997. Concerns about the lack of appropriate information on contracting 
were major elements of the 1995 and 1997 Reports of the Standing Committee of the House of Commons on 


Government Operations. In 1999, the Public Accounts Committee also expressed concerns about the information 
and recommended that future reports be audited. 


29.6 . We recommend that the Treasury Board Secretariat improve the usefulness of the Annual Contracting 
Activity Report by segregating information on amendments, standing offers and credit card payments. We also 
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recommend that the Report provide the reasons for directing a contract to a particular supplier. The Treasury 
Board Secretariat should also develop quality assurance systems for the data used to develop the Report. In 
addition, the Secretariat needs to direct attention to resolving the conceptual and presentation issues that we have 
identified. Major improvements can be achieved with some additional effort and attention to statistical reporting 
by the procurement community. 


As noted in “Subsequent Event” at the end of the chapter (paragraph 29.76), the Treasury Board Secretariat 
has already taken steps to address a number of our concerns. The Secretariat’s responses to our 
recommendations indicate that it is actively addressing some and giving what it believes to be appropriate 
consideration to the rest. 
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Fisheries and Oceans 


The Effects of Salmon Farming in British 
Columbia on the Management of Wild 
Salmon Stocks 


Chapter 30 - Main Points 


30.1 Fisheries and Oceans is managing the salmon farming industry on the basis that it poses an overall low 
risk to wild salmon and habitat. However, the Department is not fully meeting its legislative obligations under the 
Fisheries Act to protect wild Pacific salmon stocks and habitat from the effects of salmon farming. We found that 
the Department: 


e is not fully carrying out its current regulatory responsibilities to enforce the Fisheries Act with 
respect to salmon farming operations; 


e is engaged in research and is working with the Province of British Columbia (B.C.) to develop a 
regulatory framework for salmon farming, but there are shortfalls in research and monitoring to 
assess the effects of salmon farming operations; and 


e has not put in place a formal plan for managing risks and for assessing the potential cumulative 
environmental effects of proposals for new sites, should the decision be made to expand the industry. 


30.2 —‘ There is an urgent need for the Department to address these shortcomings in consultations with the 
Province if their goal of ensuring the co-existence of sustainable salmon fishing with the farming industry is to be 
achieved. The Department has recently obtained additional funds, which it plans to use to respond to industry and 
public concerns. Some projects include habitat management, further study on the environmental effects of 
aquaculture operations on fish habitat, and the effects of farmed and wild stock interaction. The Department has 
also notified the Province of B.C. and the salmon farming industry that it intends to begin assessing all fish farm 
operations to ensure that they are in compliance with the Fisheries Act. 


Background and other observations 


30.3 This is our third audit since 1997 of the Department’s Pacific salmon management programs. In previous 
chapters we reported on habitat protection and management of the salmon fisheries. We found that loss of salmon 
habitat continued and that some wild salmon stocks were declining. Salmon farming has the potential to create 
additional stress on wild salmon stocks over time, especially if the industry expands. We found that the 
Department has not developed a plan to evaluate and manage the risks in the long term. 


30.4 Reaching conclusions on the effects of salmon farming is difficult because there have been few scientific 
studies that apply directly to the B.C. situation. Recent scientific evidence of escaped Atlantic salmon reproducing 
in B.C. streams, and the potential for expansion of the industry, are raising new questions regarding effects on 
wild salmon stocks, including the cumulative effects on the environment and fisheries resources. 


30.5 Atlantic salmon are raised initially in freshwater hatcheries, and at the juvenile stage they are transferred 
to open net pens in marine coastal waters in B.C. to complete their growth. They are harvested as adult salmon. It 
is while they are being reared in open net pens that possible interactions with wild salmon and their habitat can 
occur. Furthermore, open net pens are vulnerable to damage, which can result in escapes of farmed salmon. 


The Department concurs with the contents of the chapter. 
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Fisheries and Oceans 


Fleet Management 


Chapter 31 - Main Points 


31.1 In our opinion, Fisheries and Oceans is not managing its fleet in a cost-effective manner. There is a wide 
variation in practices and procedures employed by the five regions where the fleet operations are controlled. With 
each region having its own operating practices, procedures and support, the Department is missing opportunities 
for greater flexibility in sharing resources and for better productivity in providing the fleet service. 


31.2 We observed the following problems in the key management practices that are important to a 
cost-effective fleet activity: 


e The Department has not established clear, concrete, realistic and agreed-upon performance 
expectations for the fleet. 


e The funding horizon has been only one year, even though the fleet is a capital-intensive activity with 
high fixed costs. 


e There is a lack of service accords clearly establishing the performance and funding arrangements 
between the Department’s programs and the fleet. 


e Internal budgetary processes do not support accountability for the fleet activity. 


e Information systems are not integrated and do not provide managers with reliable, timely information 
on performance and cost. 


e The method of allocating costs to programs discourages vessel use. 

e The fleet does not employ a life cycle approach to managing its vessels. 

e ©The fleet is not adequately managing its single largest operating expenditure, its human resources. 
e Shore-based support is too large, given the current size of the fleet. 

e There are weaknesses in stewardship reporting to Parliament. 


i) As the Department addresses these issues in an order of priority (they cannot all be addressed at once), it 
also needs to consider a longer-term strategy to renew its aging fleet. Such a strategy would need to take into 
consideration the changing nature of program requirements, the impact of technological change and the potential 
for alternative means of acquiring the services needed. 


Background and other observations 


31.4 At 31 March 2000, the fleet consisted of 144 vessels (122 were operational for some period during 
1999-2000), including vessels as small as an 8-metre multi-tasked utility vessel to a 111-metre heavy gulf 
icebreaker. The fleet plays a key role in providing services such as aids to navigation, icebreaking, the marine 
component of search and rescue, and marine pollution prevention and response. It also supports the Department in 
the conduct of fisheries science and enforcement, hydrography, oceanography and other marine sciences. 


31.5 In 1999-2000, Fisheries and Oceans spent approximately $229 million for operations, maintenance and 
capital replacement of the fleet. In addition, an estimated $52 million was spent on shore-based support. 
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31.6 The fleet’s human resources are its single largest operating expense. At 1 April 2000, there were 756 
officers and 1,308 crew assigned to vessels, although the number of people varies during the year depending on 
seasonal requirements. In addition, we estimate conservatively that there are about 447 full-time-equivalent 
shore-based positions that provide direct support to the fleet and about 55 positions associated with the Canadian 
Coast Guard College that provide training support directly to the fleet. 


31.7 Our audit focussed on the 47 large vessels that operated for some period during 1999-2000, vessels in 
lay-up status, and the sea- and shore-based people directly involved in managing and operating the fleet. The 
large vessels annually incur between 70 and 80 percent of the operating costs of the fleet. 


31.8 Since 1994, the fleet has had to deal with significant fiscal restraint, the merger of the former Transport 
Canada Coast Guard fleet with the science and fisheries enforcement fleets of Fisheries and Oceans (1995), and 
further fiscal restraint associated with the merger (1996 to the present). We had expected to find some 
management issues, given the amount of change the fleet has undergone; however, we are concerned about the 
number and seriousness of the issues we found. We believe that the resolution of these issues requires 
management’s prompt action. 


31.9 | While the issues raised in this chapter are considerable and represent a challenge to management, the 
Department deserves credit for several initiatives that are important to the delivery of the fleet activity. Most 
notable is the ongoing implementation of the International Management Code for the Safe Operation of Ships and 
for Pollution Prevention (ISM Code). Also, during 1999-2000, program and fleet managers identified the existing 
vessels that could be afforded, within available funding, while providing the best possible service. This exercise 
did not address the problems we identified above; however, it could bring some stability to the environment in 
which the fleet operates. 


The Department’s overall response to this chapter follows paragraph 31.139. Fisheries and Oceans accepts 
our findings and indicates that a study is under way to develop strategic options for fleet management. 
Following completion of the study, the Department intends to develop an action plan addressing seven 
specific areas that encompass the issues we have identified. 
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National Defence 


Defence Support Productivity: 
A Progress Report 


Chapter 32 - Main Points 


Rye | National Defence’s efforts since 1994 to transform itself to a more entrepreneurial organization and to 
maintain military capabilities in the face of declining financial and human resources have yet to be completed. 
Efforts to reform the services we examined in 1996 that provide support to the ships, land force brigades and air 
squadrons of the Canadian Forces remain a work-in-progress. 


32.2 In 1996 we reported that the productivity of the base/wing support functions of vehicle maintenance, 
supply and transportation had either fallen or was less than that of similar service providers. This year we found 
that, because of massive organizational and process changes, we could no longer measure productivity; nor could 
the base/wing managers tell us if they were more or less productive than they were in 1996. 


32.3 Although the number of training days delivered per Canadian Forces school employee has declined since 
1996, 14 schools may be unable to deliver the quantity of individual training required in the future. 


32.4 National Defence’s efforts to improve aircraft maintenance productivity have been relatively successful, 
although the introduction of AF 9000 Plus, a quality management system, has stalled. 


32.5 The Department has achieved limited success in its efforts to shift to a more businesslike environment in 
those renewal efforts we examined. 


e Business plans, although helpful in rationing declining resources, still lack performance measures. 
Managers continue to express concerns about the adequacy of cost information. As a result, it is 
difficult for managers to determine whether productivity is improving or declining. 


e The devolution of operating budgets to the lowest level of the organization has led to confusion over 
roles and responsibilities. Managers do not have the knowledge and experience to carry out their new 
responsibilities. The Department has not provided adequate training on new information systems, and 
the complexity of the systems has complicated rather than simplified re-engineering and devolution 
initiatives. The Department has undertaken a number of initiatives to address these issues. 


e The Department’s efforts to transform its organizational culture continue; however, culture change 
priorities have yet to be identified. 


e As might be expected, the two major alternative service delivery programs most closely associated 
with defence support productivity are meeting strong internal resistance from potentially affected 
employees. The Department expects to realize savings beginning in 2004. 


32.6 We found that the lack of measurable progress in improving support productivity can be attributed to a 
number of factors, including: 


e the relatively low priority of improving defence support productivity among the Department’s 
competing programs; 


e the need to cope with continuing personnel and budget reductions throughout the renewal period; and 


e the lack of a strategic plan to guide the Department’s reform efforts through its five-year renewal 
period (1994-1999), 
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Background and other observations 


32.7 The operational ships, land force brigades and air squadrons are sustained by a support system that 
provides such things as personnel administration, food, fuel, equipment, maintenance and training. In a force of 
about 60,000, approximately 36,000 are dedicated to operations and training at any given time, with the remainder 
focussed mainly on continuing support. In dollar terms, support services consume approximately 45 percent of the 
$11 billion budget of National Defence. In this audit, we reviewed the progress made by the Department in 
improving productivity in support functions that we audited in 1996. These functions represent only part of the 
resources consumed by support services. Measuring and improving defence support productivity is important to 
ensure that only minimal resources are consumed by support functions so that maximum resources can be 
dedicated to operations. 


32.8 In 1994, in response to the government’s 1994 Defence White Paper, National Defence embarked on a 
five-year renewal program to devote maximum resources to the Canadian Force’s combat capability by decreasing 
the cost of support activities. Between 1994 and 1999, the Canadian Forces were reduced from 76,000 to about 
60,000 members. The civilian work force was reduced from 32,000 to about 20,000 employees. During this period 
of rapid personnel reductions, National Defence completed a number of significant overseas and domestic 
deployments. 


The National Defence response is included in this chapter (paragraph 32.126). The Department stresses that 
it has faced many challenges and competing priorities that have reduced its ability to improve support 
productivity. Officials told us that business planning has resulted in greater awareness of the need for 
efficiency and that their Strategy for 2020 document will provide an overall framework for change. 


National Defence acknowledges that the development of activity-based costing information has been slow, 
but says that its decision-making process has not suffered. The Department is still working on a 
performance measurement framework. 
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Follow-up of Recommendations 
in Previous Reports 


Chapter 33 - Main Points 


33.1 This year’s reports indicate that some entities have been more successful than others in implementing 
corrective action. We noted satisfactory progress in some areas by several entities, but we also noted numerous 
areas where progress is disappointing. Some entities have made a good start in implementing changes, but have 
slowed down in completing them. Few entities get a “well done” for speedy implementation. We urge all 
departments and agencies to make a greater effort to accelerate their implementation plans. 


33-2 Our 1998 audit of selected grant and contribution programs of Industry Canada and the Department of 
Canadian Heritage identified weak management practices. Our follow-up found that Industry Canada is making 
reasonable progress in addressing our recommendations. However, we uncovered a new issue — a $20 million 
payment in advance of need resulting in as much as $5 million of interest costs to the government. We found that 
the Department of Canadian Heritage has not made sufficient improvement in the management of grants and 
contributions under its Multiculturalism Program. Program objectives remain unclear and there are still serious 
problems with due diligence. We plan to report again next year on the Department’s progress in remedying these 
problems. 


Ja3 Our follow-up of three previous chapters on Atlantic fisheries management found that Fisheries and 
Oceans is moving forward with a plan for the development and approval of a sustainable fisheries framework and 
the development of a new approach to fisheries management for the Atlantic fisheries. While the plan appears to 
address our key recommendations, the Department still needs to finalize the policy and develop detailed 
processes, procedures and legislation to support the new approach to fisheries management. Only then can we 
conclude that our recommendations have been fully addressed. 


33.4 Since our report on Population Aging and Information for Parliament, the budgetary process has 
become more transparent and forward looking. The adjustment of budget forecasts for “prudence” is now shown 
explicitly, rather than embedded in the revenue and expenditure projections. In addition, the fiscal outlook has 
been extended from two years to five years. This, however, still falls short of what is needed to show the impact of 
the impending demographic shift one to three decades ahead. 


KS BS) The departments and agencies responsible for the planning and construction of the Canadian Security 
Intelligence Service headquarters and the federal laboratories for Human and Animal Health have taken 
action on most of the original audit recommendations and observations. However, the laboratory facilities in 
Winnipeg remain significantly underutilized. The Treasury Board Secretariat has only recently initiated a review 
of its monitoring role and methodology to ensure an appropriate level of oversight over major Crown projects. 
Public Works and Government Services Canada has not resolved certain contracting issues such as the definition 
of errors and omissions in contract documents and introducing meaningful incentives for reducing costs. 


33.6 After our audit of the management of the Social Insurance Number (SIN), the government took steps to 
improve the integrity of data in the Social Insurance Register. Human Resources Development Canada has 
increased the number of investigations of SIN fraud and abuse and actions to address other issues identified by our 
audit are under way. However, a comprehensive study of the impact and extent of use of the SIN is still needed to 
fully understand the practical, economic and political implications of its use. 
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33.7. The government has made progress in its efforts to implement its Financial Information Strategy but 
considerable work is needed to ensure that the financial systems and accrual accounting policies will be in place 
by the April 2001 deadline. The longer-term integration of better financial information into day-to-day 
management decision making remains a challenge. 


33.8 In addition to those mentioned above, this chapter includes follow-up reports on the Canadian Customs 
and Revenue Agency, the Department of Finance, Human Resources Development Canada and Veterans 
Affairs Canada. Other follow-up work is reported in different chapters throughout our reports. 


a 
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Other Audit Observations 


Chapter 34 - Main Points 


34.1 The Auditor General Act requires the Auditor General to include in his Report matters of significance 
that, in his opinion, should be brought to the attention of the House of Commons. 


34.2 This chapter fulfils a special role in the Report. Our other chapters normally report on value-for-money 
audits or on audits and studies that relate to operations of the government as a whole. “Other Audit Observations” 
reports on specific matters that have come to our attention during our financial and compliance audits of the 
Public Accounts of Canada, Crown corporations and other entities, or during our value-for-money audits. 


34.3 This chapter covers the following: 
e Atomic Energy of Canada Limited records its decommissioning and site remediation liabilities; 
e managing suspected abuse and fraud in the Employment Insurance program; 
e lack of clarity on basis used in setting Employment Insurance premium rates; and 
e federal investment in Big Science: the Sudbury Neutrino Observatory. 


34.4 Although each audit observation reports matters of significance, they should not be used as a basis for 
drawing conclusions about matters not examined. 
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Sustainable Development Strategy 
for the Office of the Auditor General 


Chapter 35 - Summary 


Dock In our second sustainable development strategy, we commit to helping the federal government 
continually improve its management of environmental and sustainable development issues and to ensuring that our 
own operations are managed in an environmentally responsible manner. 


35.2 We have made good progress in implementing our first strategy, but there are several areas that require 
improvement. Our biggest challenge is to further integrate environmental and sustainable development issues into 
our audit work. We need to better document changes made to our action plan, and we must also update and 
complete the implementation of our environmental and sustainable development training plan. 


35.0 Our experience with the first strategy has allowed us to simplify our approach to managing sustainable 
development issues. We have reduced the number of goals and objectives and the indicators of performance that 
we use to measure our progress; and we have linked our actions to our planned results. From our experience with 
the first strategy, we learned that we needed to balance our environmental aspirations with other factors, such as 
increasing the quality of working life. 


35.4 During the past three years our Office, including the Commissioner of the Environment and Sustainable 
Development, has reported our findings to Parliament on subjects that include the government’s management of 
smog, toxic substances, climate change, hazardous wastes, biodiversity and sustainable fisheries. We have 
reviewed the implementation of the first round of departmental strategies. Finally, we have assessed the 
performance of departments and agencies and provided guidance on how they can improve their management of 
environmental and sustainable development issues and reduce the environmental impacts of their own operations. 


Shorts! In the next three years, we will continue to perform audits of environmental issues and to follow up on 
our previous audits. In 2001 we plan to report on an audit of the way the federal government manages its 


responsibilities for selected sustainable development issues that affect the Great Lakes and St. Lawrence River 
basin. 


35.6 Following amendments to the Auditor General Act in 1995 that formally incorporated environmental and 
sustainable development issues into our mandate, 24 federal departments and agencies tabled their first sustainable 
development strategies in the House of Commons in December 1997. Another four departments and agencies, 
including the Office of the Auditor General, voluntarily submitted strategies. The Auditor General Act requires 


that the strategies be updated every three years, and the second round of strategies was due to be tabled by 
December 2000 if Parliament were sitting. 


Bo The framework for our strategy focusses on three major areas: our audit work, our day-to-day operations, 


and our human resources. We believe it is in our audits and studies of federal organizations that we have the 
greatest impact. 


35.8 Our sustainable development strategy aims to promote sustainable development among parliamentarians, 
federal organizations and Canadians in general. More specifically, our strategy tells parliamentarians how we 


integrate environmental and sustainable development considerations into our business as a legislative audit office, 
and how we measure the success of our efforts. 


35.9 In the past three years, our Office has brought many environmental and sustainable development issues to 
the attention of Parliament and federal organizations through our financial audits, special examinations and 
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value-for-money audits, including our reviews of the sustainable development strategies. We have provided 
guidance through studies and other publications, including Moving Up the Learning Curve, a 1999 report by the 
Commissioner of the Environment and Sustainable Development. That report set out what the Commissioner 
expected to see in the second round of sustainable development strategies. 


35.10 This chapter sets out our own second strategy. It includes: 
e a description of our Office — our mandate, the work we do and the impact that we aspire to achieve; 
e asummary of our performance and the lessons learned from our first strategy; and 


e our objectives, indicators of performance and our planned results for the next three years. 
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Chapter 18 


Governance of Crown Corporations 


The audit work reported in this chapter was conducted in accordance with the legislative 
mandate, policies and practices of the Office of the Auditor General. These policies and 
practices embrace the standards recommended by the Canadian Institute of Chartered 
Accountants. The numbered paragraphs in bold face represent recommendations. 
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Governance of Crown Corporations 


Main Points 


18.1 The management of Crown corporations has improved since the Financial Administration Act was 
amended in 1984 to strengthen their control and accountability framework. Nevertheless, further improvement is 
needed in some important areas like strategic and corporate planning and the measurement and reporting of 
performance. We found a need for special attention in three areas that are central to the way Crown corporations 
are governed: 


* Boards of directors of Crown corporations need to be strengthened. They reflect Canada’s diversity 
but lack other key skills and capabilities that are needed to function effectively and to carry out their 
important responsibilities under the Financial Administration Act for the affairs of the corporation. 
Corporations need to better define their requirements for skills and capabilities and communicate 
them to the government; the government needs to act on those requirements. Boards of directors also 
need to be more engaged in the selection of their chair as well as the corporation’s chief executive 
officer (CEO). Without meaningful board involvement in the selection of the chief executive officer, 
his or her accountability to the board is weakened and corporate governance as a whole suffers. 


¢ Audit committees in Crown corporations play a crucial role in financial reporting, risk management, 
and internal control. They are the “engine” of the board of directors. Yet half of the audit committees 
we examined were operating below an effective level. Serious weaknesses in an audit committee can 
undermine the overall strength of the board. 


¢ The government has a limited capacity for reviewing and challenging Crown corporation corporate 
plans as a basis for approving them. Corporate plans set out the strategic direction of a Crown 
corporation and are intended to be the cornerstone of the Crown corporation control and 
accountability framework under the Financial Administration Act. The government needs to 
strengthen its capacity to review and challenge these plans since, once approved, they are the basis 
for holding Crown corporations accountable for conforming to government policy and for their 
overall performance. Furthermore, there is a need fer a more systematic and regular review of the 
relevance of Crown corporation mandates. 


18.2 Weaknesses in all of these areas impede the successful implementation of Part X of the Financial 
Administration Act and the quality of Crown corporation governance. They have been raised many times before, 
in Auditor General reports and other external studies and reports, but the weaknesses remain. Crown corporation 
CEOs and chairs view the resolution of these issues as critical and they need to be addressed with urgency. Other 
industrialized countries have moved ahead of Canada in these areas of Crown corporation governance. 


18.3 Corporate governance refers to the process and structure for overseeing the direction and management of 
a corporation so that it carries out its mandate and objectives effectively. It is critical that a Crown corporation, as 
a public sector body, be governed well if taxpayers’ money is to be well spent. Good corporate governance is 
important to maximize performance, prevent financial losses and help to achieve the corporate mandate. But it is 
in times of difficulty, turbulence and change that good governance is most critical. 


Background and other observations 


18.4 Crown corporations are distinct legal entities wholly owned by the government. They operate in many 
sectors, including transportation, agriculture and culture. They have more autonomy to manage than most other 
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government entities so they can operate in a more commercial manner. A board of directors oversees the 
management of each corporation and holds management responsible for the corporation’s performance. The 
government retains power and influence over Crown corporations in areas like appointments, remuneration for 
chief executive officers and directors, and approval of plans and budgets. 


18.5 Crown corporations account for a significant portion of government activity. There are currently 

41 federal Crown corporations (not including subsidiaries), employing 70,000 people. In aggregate (excluding the 
Bank of Canada), they manage $68 billion in assets and $61 billion in liabilities. While Crown corporations 
represent a significant opportunity to achieve public policy and other goals and to generate revenue, they also 
represent significant exposure to potential financial losses and other risks. 


18.6 In addition to our continuing work as auditors and examiners of most Crown corporations, we looked in 
detail at a representative group of 15 Crown corporations. We also compared Canadian Crown corporations with 
state-owned enterprises elsewhere, and used the results of other research, studies, audits and roundtables to 
support our findings. In addition, we summarized the results of special examinations — a type of value-for-money 
audit of Crown corporations — carried out between 1984 and 2000. 


In view of the fact that many of the recommendations deal with Governor in Council discretion, no detailed 
responses to the recommendations were made. Overall, the Privy Council Office states that the government 
has made strides in a number of corporate governance areas, and is committed to examining measures to 
address areas needing improvement and to ultimately strengthening governance in Crown corporations. The 
Privy Council Office also states that it is paramount to ensure that appointments to Crown corporations 
result in strong boards of directors, and that the appointment process for Crown corporations be inclusive of 
all interested parties, bearing in mind that appointments remain at the discretion of the government. 


Reaction from Crown corporation chairs and CEOs indicate that, for the most part, they agree with the 
recommendations directed specifically at them. 
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Introduction 


18.7 Crown corporations are distinct 
legal entities that are wholly owned by the 
government. Their use by government to 
deliver important public programs dates 
back to Confederation; they operate in 
many sectors of the Canadian economy, 
including transportation, energy and 
resources, agriculture and fisheries, 
financial services, culture, and 
government services. Crown corporations 
vary widely in size and in the level of 
financial support they receive from the 
government. 


18.8 Crown corporations account for a 
significant portion of the government’s 
activity. There are currently 41 Crown 
corporations (excluding subsidiaries), 
employing 70,000 people. In aggregate 
(excluding the Bank of Canada), they 
manage $68 billion in assets and 

$61 billion in liabilities. While Crown 
corporations represent a significant 
opportunity to achieve public policy and 
other goals and to generate revenue, they 
also represent significant exposure to 
potential financial losses and other risks. 
They have total debt obligations (debts 
payable to Canada and to the private 
sector) of $49 billion. Government 
budgetary appropriations are $3.8 billion 
annually. Appendix A presents a list of all 
Crown corporations (excluding 
subsidiaries) by ministerial portfolio. 


Major legislative reforms of Crown 
corporation governance in 1984 


18.9 The Financial Administration Act 
(FAA) was amended in 1984 to address 
major performance problems in Crown 
corporations. The amendments to the FAA 
(Part X) imposed a more rigorous regime, 
designed to ensure an adequate level of 
direction, control and accountability of 
Crown corporations. All Crown 
corporations fall under the FAA, except for 
some “exempt” corporations where 
Parliament agreed to create further 
distance from the government. 


Governance of Crown Corporations 


18.10 The new regime created a unique 
model for governance, establishing the 
relationship that should exist among a 
Crown corporation, the responsible 
minister, the government and Parliament. 
The regime outlines the roles and 
authorities of the corporation’s board of 
directors, management, the responsible 
minister, the Minister of Finance, the 
Treasury Board, the Governor in Council, 
Parliament, and external and internal 
auditors (see Appendix B for details). 


18.11 The regime attempts to balance 
the Crown corporation’s relationship with 
the government between its “arm’s 
length” independence in day-to-day 
activities and government’s appropriate 
direction and control. Crown corporations 
have more autonomy to manage than most 
other government entities so they can 
operate in a more commercial 
environment. A board of directors 
oversees the management of each 
corporation and holds management 
responsible for the company’s 
performance. Through the Crown 
corporation’s chair, the board of directors 
is accountable to a minister who 
represents the government and acts as the 
link between the corporation and both 
Cabinet and Parliament. The government 
retains power and influence over Crown 
corporations in areas like appointment and 
remuneration of directors and chief 
executive officers, directives and 
regulations, and approval of corporate 
plans and budgets. Each Crown 
corporation is accountable to Parliament 
through the responsible minister. 


18.12 The FAA amendments also 
strengthened internal audit and introduced 
special examinations (a type of 
value-for-money audit) in most Crown 
corporations, in addition to the annual 
audits of financial statements. Special 
examinations provide the board with an 
independent opinion on how well the 
corporation is being managed. A Crown 
corporation must undergo a special 
examination every five years; the 
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Crown corporations 
account for a 
significant portion of 
the government’s 
assets, liabilities and 
expenditures. 


Crown corporations 
have more autonomy 
to manage than most 
other government 
entities. 
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EE 
Governance of Crown Corporations 


Research shows a 
relationship between 
good governance and 
good results. 


period 1996 to 2001 represents the third 
cycle of special examinations since the 
FAA was amended. Appendix C outlines 
the role of special examinations and 
provides more detail on what they 
examine. It also presents an overall 
summary of findings over the three 
examination cycles to date. 


18.13 Each Crown corporation’s 
enabling legislation, whether a special act 
of Parliament or articles of incorporation 
under the Canada Business Corporations 
Act, sets out in broad terms its mandate, 
powers and objectives. 


Good corporate governance is more 
important than ever 


18.14 Like most organizations, Crown 
corporations face a complex and rapidly 
changing environment. The private sector 
is increasingly active in areas that once 
were the exclusive domain of Crown 
corporations. Technological advances and 
consumer demands for new and different 
services create pressure for change. 


18.15 Crown corporations must deal 
with the added complexity of managing in 
the public sector. Corporations in the 
private sector operate to maximize 
shareholder value. However, the 
objectives of Crown corporations are not 
always that clear. Many are required to 
achieve financial self-sufficiency while 
meeting public policy objectives (such as 
delivering needed services that are not 
commercially viable). 


18.16 Corporate governance refers to 
the process and structure for overseeing 
the direction and management of a 
corporation so that it carries out its 
mandate and objectives effectively. It is 
critical that a Crown corporation, as a 
public sector body, be governed well if 
taxpayers’ money is to be well spent. 


18.17 Good corporate governance is 
essential if a Crown corporation is to fulfil 
its mandate. In good times, good 
governance can increase the effectiveness 


of performance and help to prevent 
potential financial losses, ineffectiveness 
and inefficiencies. But it is in times of 
difficulty, turbulence and change that 
good governance is most critical. 


18.18 In the recent past, landmark 
studies (for example, the Cadbury Report, 
United Kingdom 1992, Gérard Veilleux’s 
report “Unfinished Business” in 1993, and 
the 1994 Toronto Stock Exchange 
Guidelines for Improved Corporate 
Governance in Canada) have focussed on 
the need to address gaps in corporate 
governance. Corporate failures and 
scandals have been linked to weaknesses 
in governance. The Canadian Institute of 
Chartered Accountants, supported by the 
Toronto Stock Exchange and the Canadian 
Venture Exchange, has established a Joint 
Committee on Corporate Governance to 
review the current state of corporate 
governance in Canada, compare Canadian 
practices with international best practices, 
and recommend changes to ensure that 
Canadian corporate governance is among 
the best in the world. It has become an 
accepted reality for private and public 
sector corporations that good governance 
is essential to long-term success. 


18.19 The literature shows a 
relationship between good governance and 
good results. For example, companies 
whose standards of corporate governance 
are high are more likely to gain the 
confidence of investors and support for the 
development of their businesses. 


Focus of the audit 


18.20 Our special examinations over 
the last 15 years indicate that overall the 
management of Crown corporations has 
improved since the FAA was amended. 
Nevertheless, further improvement is 
needed in some important areas like 
strategic and corporate planning and the 
measurement and reporting of corporate 
performance. 


18.21 Government documents, 
important studies, professional literature 
and the accountability and control regime 
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for Crown corporations all point to the 
fundamental importance of effective 
corporate governance. However, our 
experience with auditing Crown 
corporations suggests a need for special 
attention to three areas that are central to 
the governance of Crown corporations: 


* the process of appointing the chief 
executive officer (CEO) and the board 
chair and directors; 


+ the composition, role, 
responsibilities, and performance of the 
board’s audit committee; and 


+ the government’s capacity to review 
and challenge the corporation’s corporate 
plan before approval and to ensure that 
Crown corporation mandates continue to 
be relevant. 


18.22 Weaknesses in these three areas 
impede the successful implementation of 
Part X of the FAA and the effective 
governance of Crown corporations. 
Previous reports of the Auditor General 
have raised the need to resolve them, as 
have studies and reports by others, but the 
weaknesses remain. This chapter discusses 
them in detail. Crown corporation CEOs 
and board chairs cite as an ongoing 
frustration these continuing weaknesses in 
areas they view as high priorities. Other 
countries have moved ahead of Canada in 
these areas of Crown corporation 
governance. 


18.23 Given their importance to good 
governance of Crown corporations and the 


Governance of Crown Corporations 


Observations and 
Recommendations 


Appointing Boards of Directors, 
Board Chairs, and Chief Executive 
Officers 


Boards of directors have a pivotal role 


18.24 The government has stated, 
“...Crown corporations will operate at 
peak efficiency only when boards of 
directors operate at peak efficiency” 
(Crown Corporations Direction, Control, 
Accountability, Government of Canada’s 
Proposals, Privy Council Office, 1977). 
The FAA stipulates that the board of 
directors is responsible for the affairs of 
the corporation. The Treasury Board’s 
guidelines on Corporate Governance in 
Crown Corporations and Other Public 
Enterprises (1996) include a number of 
ways to strengthen the board’s 
effectiveness. The guidelines state that a 
board of directors has a duty to oversee 
the management of its Crown corporation 
with a view to both the best interests of 
the corporation and the long-term interests 
of the government. To fulfil that 
stewardship duty, the board is expected to 
exercise judgment in establishing the 
corporation’s strategic direction, 
safeguarding the corporation’s resources, 
monitoring corporate performance, and 
reporting to the Crown. The board is 
accountable for its performance to the 
responsible minister, and the minister is 
accountable in turn to Parliament for the 
performance of the corporation. 


It is in the 
government's interest 
to appoint strong 
boards of directors to 
oversee the 


concerns identified during the course of management of Crown 
our es audits, these three areas were 18.25 It is therefore in the government’s corporations. 

the focus of this audit. We set out to interest to appoint strong boards of 

examine how well these areas were now directors. The appointment of directors is 

functioning. The audit scope included key —_a fundamental prerogative of the 

provisions of Part X of the FAA and shareholder in private sector and public 

parallel provisions in the enabling sector corporations, including Crown 

legislation of Crown corporations corporations. Under the FAA, the 

exempted from Part X. Further details on responsible minister appoints directors 

the audit scope, objectives and criteria are with the approval of the Governor in 

presented at the end of the chapter in Council, following a review of the 

About the Audit. minister’s recommendations by the Prime 
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Governance of Crown Corporations 


Many chairs and CEOs 
are not satisfied with 
the mix of skills and 
capabilities on their 
boards. 


Exhibit 18.1 


Comments on Board 
Skills and Capabilities 
by Chairs and CEOs 


Minister’s Office. The government needs 
to appoint strong boards, allow them to do 
their work, and dismiss those that do not. 
Given the pivotal role of the board of 
directors in governing a Crown 
corporation, we looked for timely 
appointments of qualified directors who 
met the requirements of both the 
government and the corporation. 


Boards reflect Canada’s diversity but 
lack other key skills and capabilities 


18.26 The first step in any effective 
appointment process is to identify the 
skills and capabilities the position 
requires. The government has made a 
commitment to appoint, on the basis of 
competence and equity, directors who 
meet the following requirements: 


« Appointees must meet the criteria 
that may be specified in the corporation’s 
particular enabling legislation. 


¢ Geographic regions must be well 
represented on the board and geographic 
concerns taken into account. 


* The capacity to speak both official 
languages is often required. 


+ Persons appointed will better 
represent women, visible minorities, 
Aboriginal peoples, and people with 
disabilities (So You Want An 
Order-In-Council Appointment? by 
Penny Collenette, Prime Minister’s 
Office). 


18.27 The government has improved 
the gender balance and geographic 
representation on boards of directors. 
According to the Conference Board of 
Canada, Crown corporation boards 
continue to lead all sectors in board 
gender balance, with women accounting 
for 30 percent of directors, compared with 
10 percent in other sectors; and 18 percent 
of board chairs are women, compared with 
6 percent in other sectors. Sixty-five 
percent of Crown corporations view 
geographic representation as the top 
criterion in the selection of new board 
directors, compared with 25 percent of 
private sector corporations. 


18.28 Many Crown corporation chairs 
and CEOs report that they are not satisfied 
with the mix of skills and capabilities on 
their boards. According to the Conference 
Board of Canada, 18 percent of Crown 
corporations view a specific skill set to 
complement the board as the top criterion 
in the selection of new board directors, 
compared with 58 percent of private 
sector corporations. When we asked 
Crown corporation chairs and CEOs what 
necessary capabilities or skills their boards 
of directors were missing, they noted gaps 
in one or more of financial expertise and 
financial literacy, marketing, accounting, 
strategic leadership, previous experience 
on a board of directors or related 
experience, ideally in an organization of 
similar stature (assets, revenues, 
complexity), and knowledge of the 
corporation’s business. They also noted 
the value of having a chair or one or more 


“Politics, gender and geography are the top three selection criteria now....Relevant business 


experience needs to be number one.” 


“Twenty-five percent of the board today are out of their depth due to political criteria being used 


[for selection] instead of business.” 


“Depth of experience is needed, and character to promote a close working relationship with the 


CEO” 


“Experience in specific industries (for example, financial services) is essential to the board, but 


ignored in recent appointments.” 


“We have lots of lawyers, which is fine, but could use at least one accountant.” 
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directors with previous CEO experience. 
Exhibit 18.1 presents some of their 
comments. 


18.29 We also asked chairs and CEOs 
about the impact of these gaps in skills. 
Their responses indicate that the gaps 
undermine the board’s effectiveness. For 
example: 


Today, the only individual on our 
board who has a strong financial 
background is the CEO....How can 
the board direct, oversee or even 
support the corporation’s work 
effectively? 


CEOs look to the board, and 
individual directors, to run ideas by, 
and to engage in in-depth 
exploration....This role of the board 
can’t be fulfilled now. 


18.30 These gaps in board skills and 
capabilities are a reflection of the 
appointment process, which we examined 
in some depth. The selection process for 
board directors must seek out people with 
the capabilities and skills that are essential 
to the effective functioning of the board; 
that is in the interest of both the 
government and the Crown corporation. 
The government needs to understand the 
skills and capabilities the corporation 
needs and lacks on the board when it 
looks for candidates to fill board 
vacancies. At the same time, it also needs 
to meet its own objectives. The final 
appointment must reflect the needs of both 
the government and the corporation. 


Appointments need to meet the board’s 
skill and capability requirements 


18.31 Both the government and the 
Crown corporation have a role to play in 
ensuring that directors have the necessary 
qualifications to meet the board’s needs. 
Under the FAA, the responsible minister 
appoints the board of directors with the 
approval of the Governor in Council. That 
the “board of directors should contribute 
to its own renewal” is also clearly stated 


Governance of Crown Corporations 


in the Treasury Board’s guidelines on 
corporate governance: 


The selection and recruitment of 
knowledgeable and skilled candidates 
to the board is of prime 
importance....Recommendations by 
the existing board influence the 
selection and approval process. The 
board’s experience...provides an ideal 
basis for the board to review the 
suitability of its composition and the 
effectiveness of its performance.... 
The board’s self-assessment of the 
skills required can be a useful basis 
for recommendations on the 
appointment or replacement of 
directors as terms approach expiry. 
The chair, on behalf of the board, 
should advise the appropriate minister 
and the Director of Appointments in 
the Prime Minister’s Office of the 
desired mix of skills useful for the 
board and, in particular, those skills 
that should be sought in upcoming 
vacancies. 


18.32 Consistent with best practices, 
boards need to produce profiles that 
specify the skills they require based on 
identified needs or gaps, and that focus on 
the positions themselves rather than the 
individuals who might fill them. The 
board profiles must capture the skills the 
corporation will need in the future to 
move its strategic direction forward. In 
his 1993 report, Gérard Veilleux 
recommended the use of board profiles in 
Crown corporations. The Privy Council 
Office, the Treasury Board, the Auditor 
General, and several provincial 
governments endorsed the 
recommendation. 


18.33. And yet we found that many 
boards of directors have not developed 
such profiles. In 1995, we reported that 
13 percent of Crown corporations were 
using board profiles and we encouraged 
ministers to lend their support to the 
practice. We find now that 34 percent of 
all corporations have completed profiles 
or selection criteria for board members. 
Typically, the Crown corporation sends 
these to the minister’s office, and only 


Gaps in skills and 


capabilities undermine 


the board’s 
effectiveness. 


Only 34 percent of 


Crown corporations 


have completed 


profiles outlining their 


requirements for 
director skills and 
capabilities. 
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Where the government 
acted on board skills 
profiles, the 
appointment better 
met the board’s needs. 


sometimes to the Privy Council Office or 
the Prime Minister’s Office. 


18.34 Job descriptions for the board of 
directors, the chair and the CEO can 
provide a good context for developing the 
board skills profile. The corporations that 
have prepared job descriptions have found 
them very useful. For example, the 
process itself builds a shared 
understanding and communication among 
the players on their respective roles, and 
the job description helps to outline the 
nature of the commitment expected when 
recruiting or orienting new board 
directors. 


18.35 Fewer than half the corporations 
that do use board skill profiles have found 
them effective in the selection of 
directors. Many reported that while they 
can and do suggest what they need in 
future appointments, they have no 
assurance that the government takes their 
suggestions seriously; many said that the 
government has not used their profiles at 
all. 


18.36 We reviewed a sample of board 
appointments to better understand the 
process. At one extreme, the government 
made the appointment after consulting 
only nominally with the board chairman, 
or not at all. In many cases, the 
corporations had prepared skills profiles 
but found that the government did not use 
them. 


18.37 At the other extreme, the 
corporation’s board of directors conducted 
the search and recruitment and 
recommended a single name to the 
government for approval. In some of these 
cases, the chair vetted the list of potential 
candidates for those who met the 
shareholder’s requirements. 


18.38 In between these two extremes, 
the government led the search and 
actively consulted with the board. The 
corporations developed board profiles that 
they believed the government used. In 
these cases, the chair had a good 
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relationship with the minister, a factor that 
appeared to smooth the appointment 
process. 


18.39 We found that meetings between 
the key players were also valuable in 
communicating the board’s desired mix of 
skills and capabilities that should be 
sought in filling upcoming vacancies. For 
example, some board chairs took the 
initiative to meet with the Prime 
Minister’s Office and the Privy Council 
Office in advance of a board vacancy to 
discuss the needs of the board, in keeping 
with the Treasury Board guidelines. 


18.40 Overall, we found that where the 
government acted on the board’s profiles 
and stated requirements, the appointment 
better met the board’s needs. For example, 
in some situations the minister sent the 
resumés of possible candidates to the 
corporation for consideration and acted on 
the corporation’s assessment of them 
when selecting board directors. Too often, 
however, the corporation had not assessed 
its requirements for skills and capabilities 
and the government did not consult the 
board, which led to frustration, unmet 
needs, and a weakened board. 


18.41 Other industrialized countries are 
ahead of Canada in their practices of 
appointing board directors. In New 
Zealand, for example, the objective in 
selecting directors is to ensure that the 
board has the necessary skills to enhance 
the corporation’s performance and the 
interaction and operations of the board, 
and to ensure that the appointment is 
consistent with the corporation’s strategic 
direction and its needs. The process 
reinforces the minister’s role in 
identifying the skills needed in a 
particular position on a Crown corporation 
board, appointing a suitable candidate, 
and reviewing the performance of the 
directors and the board. It begins with 
agreement between the minister and the 
corporation on a board skills profile and it 
ends with the appointment of the selected 
candidate, after consultation and/or 
interviews with the corporation, the 
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responsible minister and the Minister of 
Finance. This is followed by the induction 
of the appointee. New Zealand publishes 
an outline of the key competencies 
required in directors. For example, 
preference is given to candidates who 
have experience in one or more of the 
following: 


* governance in a significant 
organization with a commercial focus; 


* chief executive or senior 
management positions in an organization 
with commercial attributes; 


* senior positions in relevant 
professional disciplines, such as science, 
technology, finance, law, health, 
agriculture, and social policy; and 


« related governance or management 
positions in community or professional 
organizations. 


18.42 The United Kingdom has a 
Commissioner for Public Appointments, 
whose objective is to manage an 
appointment process that will merit the 
confidence of candidates and the public. 
The government requires that the process 
be efficient, transparent, and based on 
merit. The process incorporates a number 
of procedures and principles such as 
scrutiny by an independent assessor, 
transparency and openness, and 
proportionality (which means the 
appointment process is appropriate to the 
nature of the post and the magnitude of its 
responsibilities). All stages of the process 
are subject to audit. No one is appointed 
to a public body without some form of 
interview, and candidates must answer a 
question about their political activities, to 
provide transparency and to identify 
related skills and experience that could be 
useful and that may demonstrate 
commitment. The government states that 
political activity and affiliation are 
normally not criteria for appointment. 


18.43 Two recently created Crown 
corporations in Canada have adopted the 
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practice of appointing a nominating 
committee to identify director candidates, 
a model that could be used more widely. 


18.44 Each Crown corporation should 
develop a board skills profile. The chair, 
on behalf of the board, should 
communicate the profile to the 
responsible minister, the Privy Council 
Office and the Prime Minister’s Office, 
as well as the board’s specific skills and 
capability requirements for upcoming 
vacancies. The government should act 
on these stated requirements in its 
selection of directors. 


18.45 Each responsible minister and 
the corporation should reach an 
understanding on how the board will be 
engaged in the selection and 
appointment of directors. 


A need to maintain continuity and 
experience on boards of directors 


18.46 We analyzed appointments made 
to 15 Crown corporation boards 

from 1996 through 2000. We drew our 
data from appointment guides published 
by the Privy Council Office, outlining the 
terms and conditions for all appointments 
approved by the Governor in Council. 


18.47 The FAA states that directors can 
be appointed to terms of up to three years, 
and reappointed if desired. It provides for 
directors to continue serving after their 
terms have expired until either they are 
reappointed or a new appointment is 
made. This provision ensures that boards 
can continue to function, even if there are 
delays in the appointment process. 


18.48 The length of time directors 
serve is improving, but still too short. 
Most directors of Crown corporations are 
appointed to the maximum three-year 
term allowed for by the FAA, although 
there have been some terms of one and 
two years. The biggest factor in the length 
of time a director serves on the board is 
not the length of the term but rather the 
number of reappointments. There are 
positive signs that indicate some 


More directors are 
serving on Crown 
corporation boards for 
longer periods. 
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There is a need for 
better director training. 


improvement. More directors are serving 
on Crown corporation boards for longer 
periods, which means that the number of 
reappointments is increasing. 

Sixty-four percent of incumbents eligible 
for reappointment have been reappointed 
for at least a second term. This higher rate 
of reappointment has helped to increase 
the average duration of service by 
directors, from 3.9 years in 1997 to 

4.6 years in 2000. However, length of 
service still lags behind the private sector 
where, according to the Conference Board 
of Canada, directors serve an average of 
7.7 years. 


18.49 Appointment decisions are not 
timely. Crown corporations report that a 
serious problem impacting on board 
continuity and stability is the length of 
time it takes to decide on many 
appointments. The amount of time from 
the date a director’s term expires to the 
date of reappointment or appointment of a 
new director is called the “expired 
period.” 


18.50 We examined 79 appointments 
whose terms had expired over the last 
five years. While in 2000 the average 
expired period was five months, in 

70 percent of the corporations in our 
sample it was longer than six months — 
and longer than a year in half of them. In 
one Crown corporation, 80 percent of the 
directors have continued in expired 
positions for over a year, and the chair 
position is vacant. 


18.51 Appointment terms are 
unevenly staggered. In 6 of the 15 Crown 
corporations in our sample, the current 
terms of more than 50 percent of the board 
directors will expire in a single year — 
and as many as 80 percent in two 
corporations. The FAA requires that as far 
as possible no more than 50 percent of a 
board’s director positions should expire in 
any one year. Coupled with the slow 
process of deciding on appointments, this 
kind of turnover leaves many boards with 
considerable uncertainty about their 
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composition. Expiry dates need to be 
staggered evenly to ensure that continuity 
of expertise and corporate memory is not 
compromised. 


18.52 The government should decide 
on Crown corporation director 
appointments on a timely basis, improve 
the staggering of term expiry dates and 
increase the length of service of 
qualified directors. 


Other factors affect board functioning 


18.53 Past reports of the Auditor 
General have observed that new directors 
were not adequately briefed on their 
duties. Despite some improvement, the 
situation today is similar. Crown 
corporations generally provide orientation 
sessions for new directors, and some urge 
their directors to attend external training 
sessions. However, there is a need for 
training in the director’s responsibilities to 
the corporation, the corporation’s 
relationship with the government, 
compensation policies for Crown 
corporation executives, and board 
procedures. There is a need to 
professionalize the role of directors, many 
of whom have had no previous experience 
on boards of any kind. 


18.54 The government, with Crown 
corporations, should ensure that newly 
appointed directors are provided with 
adequate orientation and training in 
their responsibilities to the corporation, 
the corporation’s relationship with the 
government, compensation policies for 
Crown corporation executives, and 
board procedures. 


18.55 Senior government officials and 
some Crown corporations have raised 
questions and concerns about the 
appointment of public servants, 
particularly deputy ministers of federal 
government departments, to boards of 
directors of Crown corporations. These 
appointments pose particular challenges 
for the individuals as well as for the 
boards on which they serve. The Treasury 
Board guidelines point out that the 
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knowledge and expertise of deputy 
ministers can assist the board in better 
appreciating government policy. Similarly, 
participation as a member of the board of 
directors improves the deputy minister’s 
understanding of the corporation. Crown 
corporations have raised other issues 
about public servants on their boards of 
directors, including attendance records 
(particularly for deputy ministers), a 
concern that some boards include too 
many public servants, and a practice by 
some deputy ministers of periodically 
sending a substitute to a board meeting. 
The deputy minister sits on the board not 
as a source of direction from the minister 
but with the same statutory obligations as 
other directors. Yet some corporations 
report that other directors sometimes defer 
to the deputy minister because they 
believe he or she is a direct spokesperson 
for the government. Some are also 
concerned about the potential for conflict 
between the individual’s dual roles as 
director and as deputy minister. It is 
important that deputy ministers be 
particularly sensitive to these issues. The 
role and attendance of public servants on 
Crown corporation boards of directors 
needs ongoing, close monitoring. 


Chair appointments often made with 
limited board consultation 


18.56 The chair of a Crown corporation 
acts as the formal link between the 
minister and the corporation. The chair 
manages the affairs of the board of 
directors and is not a part of the 
corporation’s management. 


18.57 Under the FAA, the Governor in 
Council has the authority to appoint and 
fire the chair, and to fix the compensation. 
The Act also provides for the board to 
influence the appointment, through a 
provision that requires the responsible 
minister to consult with it. The chairs, 
CEOs and directors we interviewed said 
that the board needs to play an active role 
in the chair’s appointment, including 
assessing board needs and profiling skills. 
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We expected to find that the government 
consulted meaningfully with the board as 
a whole. 


18.58 Many of the corporations in our 

sample report that there has been limited 

consultation with the board, if any, on the 
selection and appointment of the chair. 


18.59 The government should ensure 
that it consults with boards of directors 
of Crown corporations on chair 
selection and appointment. 


Appointment of the chief executive 
officer is a key factor in corporate 
governance 


18.60 The key to good management of 
a corporation is the chief executive officer 
(CEO). The FAA gives the Governor in 
Council the authority to appoint and fire 
the CEO and to fix remuneration. The Act 
also provides for the board to influence 
the CEO’s appointment, in requiring that 
the responsible minister consult with the 
board. The board has the authority to 
evaluate the CEO’s performance. 


18.61 The Treasury Board guidelines 
on corporate governance in Crown 
corporations stress the importance of the 
relationship between the CEO and the 
board: 


There must be a 
clear accountability 
relationship between 
the CEO and the 
board. 


In a Crown corporation, the nature of 
the relationship between the CEO and 
the board is often critical. The board 
must work with the CEO to build a 
relationship of openness and trust.... 
An important aspect...is to establish a 
clear accountability relationship for 
the CEO to the board.... 


18.62 In the private sector, this 
accountability relationship is achieved by 
virtue of the board’s power to hire and fire 
the CEO and fix remuneration. Those 
powers are critical to good governance in 
any corporation. But some have pointed to 
an accountability dilemma in Crown 
corporations. After far-reaching 
consultation and research, the Public 
Policy Forum’s 1998 study concluded, 
“The lack of hire, fire and pay powers 
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Many Crown 
corporation CEOs are 
quite candid in seeing 
that their essential 
accountability lies with 
the government and 
not with the board. 


over the president deprives Crown boards 
of the most powerful accountability lever 
exercised by private sector boards.” The 
Public Policy Forum report added that 
many Crown corporation CEOs do not 
feel accountable to their boards. It said, 
“Many Crown presidents, while paying lip 
service to the valuable advice they receive 
from their boards, are quite candid in 
seeing that their essential accountability 
lies with the government and particularly 
with those at the centre of it.” 


18.63 Research by the Conference 
Board of Canada indicates that 90 percent 
of Crown corporations hold the view that 
boards must be involved in the final 
selection of the CEO. This is perceived as 
the most significant issue in Crown 
corporation governance. Asked what one 
thing they would change in the selection 
and appointment process, Crown 
corporations overwhelmingly said they 
would increase the board’s involvement in 
the selection of directors and the CEO. 


18.64 In our view, the CEO 
appointment process ought to be measured 
by more than the strength of the CEO 
selected. The appointment also needs to 
reinforce the CEO’s accountability 
relationship with the board, while 
preserving the appointment prerogative of 
the Governor in Council. For example, if a 
strong CEO is selected but the chair and 
the board lack confidence in the process 
that led to that selection, then the single 
most important relationship in the 
corporation — between the chair and the 
CEO — can begin with doubts and a lack 
of trust. 


18.65 We noted the use of three 
different models in recent appointments of 
CEOs to Canada’s Crown corporations. 
The models varied in approach, as did the 
way each applied the requirement for 
“consultation with the board”: 


- “Centre selects” model. The 
identifying feature of this model was that 
the government not only controlled the 


search for the new CEO but also made the 
selection without meaningful consultation 
and buy-in from the board and the chair. 
In effect, the corporation was told who the 
CEO would be. 


- “Centre searches” model. In this 
model, the government ran the CEO 
search and recruitment process, but made 
the selection only after meaningful 
consultation with and buy-in from the 
chair as proxy for the board. The chair 
was a member of the search committee, 
along with representatives of the Privy 
Council Office (PCO), the Prime 
Minister’s Office (PMO) and the 
responsible minister. The search 
committee established the selection 
criteria, in some cases using the board’s 
input, and conducted the interviews. The 
board had no direct role, but in some cases 
the chair went back to the board for input 
on the shortlist of candidates. The PMO 
advertised the position and accepted 
candidates’ applications. The search 
committee conducted the initial screening 
of applicants. The PMO confirmed the 
selection of the candidate recommended 
by the search committee. 


¢ “Board searches” model. Under this 
model, the Crown corporation led the 
CEO search process through either a 
search committee or the board. The board 
asked the PCO and the PMO to suggest 
any additional candidates, and 
recommended a shortlist to the 
government. Candidates were interviewed 
by the board and then by the minister. The 
government made the final selection of 
the CEO. 


18.66 The literature and best practices 
show conclusively that active and 
independent board governance leads to 
stronger corporate performance. When the 
board has no meaningful involvement in 
the selection of the CEO, problems of 
accountability can arise between them. In 
many industrialized countries, boards play 
a leading role in the search for their CEO, 
not just in private sector firms but in 
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state-owned enterprises that are 
comparable with Crown corporations. 


18.67 Our interviews with a sample of 
chairs, CEOs and directors of the Crown 
corporations involved in the three CEO 
appointment models we have described 
confirmed that they strongly and 
consistently favour a greater role for the 
board in the CEO search and selection 
process. They rated the three appointment 
models in terms of governance 
effectiveness. 


“Centre selects” model found least 
favour 


18.68 The Crown corporation leaders 
we interviewed, including (perhaps 
surprisingly) the CEOs themselves, rated 
the “centre selects” model the worst of the 
three by far. It is also the model furthest 
from good governance practices. Chairs 
and directors whose CEO was selected by 
the “centre selects” model said 
unequivocally that the exclusion of their 
chair and board members in the CEO’s 
selection had led to serious governance 
problems. 


18.69 CEOs selected by this process 
quickly understand that their true 
accountability is to the government and 
not to their board. They acknowledge that 
they gain access and influence with the 
centre of government when the 
government alone drives the selection 
process. However, this very access 
compromises the board’s capacity to 
govern. Lack of government consultation 
with the board on the CEO selection often 
amounts to lack of buy-in and the creation 
of a rift between the board and the CEO 
before he or she even takes office. Our 
interviews with Crown corporation leaders 
indicate that the CEO must perceive that 
the board has real power and is leading the 
selection process if the accountability 
relationship between the CEO and the 
board is to be effective. 
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“Centre searches” model got mixed 
reviews 


18.70 Some see the “centre searches” as 
the new model for CEO selection. 
However, while board chairs and the 
government viewed it favourably, we 
found that CEOs and directors were less 
supportive. The model has some definite 
strengths. The board chair is on the search 
committee along with the other key 
players — the Prime Minister’s Office, the 
minister’s office, and the Privy Council 
Office; the PCO and the PMO have solid 
experience in carrying out a search 
process. This model also helps to build 
relationships among the players. 


18.71 The model has some fundamental 
weaknesses, however. First, board 
directors state that they have had no 
meaningful involvement in the CEO’s 
selection under this model. The Crown 
corporation’s involvement through the 
chair as sole proxy, while an improvement 
over the “centre selects” model, is still 
limited. This model does not effectively 
engage the board as a whole in the search 
process. As a result, the board may be less 
committed to the selected candidate. 
Second — and this is key — the 
candidates see that the PCO, the PMO and 
the minister’s office dominate the process: 
they advertise for candidates and 
candidates respond to them. Third, three 
of the four participants on the search 
committee represent the government, 
giving it the balance of power in selecting 
the candidate. Overall, the “centre 
searches” model does not go far enough 
and we would view it as a transitional 
model. 


CEOs, chairs and 
directors see the 
“board searches” 
model as supporting 
good governance. 


“Board searches” model seen as the 
most effective 


18.72 The model rated the most 
effective by chairs, CEOs and directors 
was the “board searches” model. That 
they see it as supporting good governance 
and a strong accountability relationship 
between the board and the CEO is clear 
from the following comments: 
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The government needs 
to move toward a 
process in which the 
Governor in Council 
selects the CEO from 
board-recommended 
candidates. 


It worked, it included all levels in the 
process... 


If the board is to fulfil its mandate, it 
must be given a major role in CEO 
recruitment. 


Excellent model not only in terms of 
the result but also for board buy-in 
and the support of the new CEO. The 
most important role of a board is to 
hire, retain, evaluate and compensate 
the CEO — often taken out of our 
hands by the shareholder. 


18.73 The “board searches” model is 
the most consistent with best practices in 
comparable organizations. Also, chairs, 
CEOs and directors have reported strong 
mutual trust and respect among all 
corporate players and with the government 
when this model is used. 


18.74 The enabling legislation of some 
Crown corporations includes a “‘board 
searches” model. It gives the corporation 
the statutory authority to appoint the chief 
executive officer, with the approval of the 
Governor in Council. 


A transition strategy may be necessary 


18.75 It has been argued that at present, 
some Crown corporation boards are not 
strong enough to conduct the search for a 
CEO along the lines of the “board 
searches” model. We recognize that many 
boards lack key capabilities and, as a 
result, difficulties have sometimes 
developed when the “board searches” 
model has been used. In our view, the 
solution is for the government to move 
toward a “board searches” model in which 
the Governor in Council selects the CEO 
from a slate of board-recommended 
candidates as the government undertakes 
to strengthen Crown corporation boards. 


18.76 The move to a “board searches” 
model calls for a transitional strategy that 
could be adapted to meet the particular 
needs of each Crown corporation. The 
extent of government support and 
involvement or external expertise could 
vary, depending on a board’s capability at 
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any given time; an interim approach could 
be used to involve-the board more directly 
as it develops the capacity to carry out a 
“board searches” model. For example, the 
“centre searches” model could be adapted 
to include members of the board’s search 
committee along with the chair and 
representatives of the PMO, the PCO, and 
the minister’s office. Or a committee of 
the board could interview and make 
recommendations on a shortlist of 
candidates selected by the chair, the PMO, 
the PCO, and the minister’s 
representative. In addition, boards may 
require some training in how to conduct 
an effective CEO search process. 


18.77. The government remains 
ultimately accountable for the 
performance of a Crown corporation, and 
the FAA gives the Governor in Council the 
authority to appoint, fire and remunerate 
the CEO. None of this changes with a 
“board searches” model; the difference 
lies in who is seen by candidates and the 
corporation to be leading the process. The 
government would need to work closely 
with the corporation and remain directly 
involved at each stage of the process, 
contributing selection criteria, for 
instance, and suggesting potential 
candidates. 


18.78 The board of directors of a 
Crown corporation, in direct 
consultation with the minister, the 
Prime Minister’s Office and the Privy 
Council Office, should lead the process 
of selecting the corporation’s chief 
executive officer for approval by the 
Governor in Council. A transition 
strategy should be used where a board 
does not yet have the capability to carry 
out this approach. 


18.79 In each case, the responsible 
minister, the Crown corporation, the 
Prime Minister’s Office and the Privy 
Council Office should reach an 
understanding on the respective roles of 
the board and the government in 
selecting and appointing the CEO. 
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Committee 


18.80 The audit committee is a core 
committee of a corporation’s board of 
directors. It represents the “engine” of a 
well-functioning board. Given the rapidly 
increasing scope and importance of its 
role, the audit committee’s effectiveness is 
attracting greater attention in the 
corporate world. 


18.81 An audit committee that operates 
effectively can bring significant benefits 
to the corporation. It can help to: 


* improve the quality of financial 
reporting; 


* ensure that key corporate risks are 
identified and managed adequately; 


* strengthen the internal audit 
function; 


* enable the directors to contribute 
their independent judgment and play a 
positive role in overseeing the 
corporation’s business operations; 


+ facilitate better communication 
among management, directors and internal 
and external auditors; 


+ reinforce the independence of the 
internal and external auditors; 


* create a climate of discipline and 
control that will reduce the opportunity for 
fraud: 


* increase stakeholder confidence in 
the credibility and objectivity of corporate 
performance reports; and 


* obtain greater assurance that assets 
are protected and resources are managed 
economically, efficiently and effectively. 


18.82 The FAA specifically charges 
each parent Crown corporation to 
establish an audit committee. The 
committee’s legislated responsibilities are 
to oversee internal audit; review and 
advise the board of directors on the 
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financial statements, the annual auditor’s 
report and the special examination plan 
and report; and perform other functions 
delegated to it by the board. However, the 
audit committee usually assumes 
oversight responsibilities well beyond 
these duties. It is expected to oversee the 
management of a wide range of financial 
and non-financial risks that affect the 
corporation. 


18.83. We looked at the performance of 
14 Crown corporation audit committees. 
We compared their practices with a dozen 
recognized best practices (see 

Exhibit 18.2), to provide a basis for future 
self-assessment and to identify potential 
for improvement in their performance. We 
used a combination of document review 
and our own observations to assess their 
effectiveness, given that the Auditor 
General is the statutory auditor of most 
Crown corporations (the complete list of 
best practices is provided in Appendix D). 


18.84 We expected that audit 
committees would have the appropriate 
skills and experience to carry out their 
role and their duties effectively. Because 
the best practices we selected for 
comparison are viewed as key practices 
and are consistent with existing FAA 
requirements, we expected that Crown 
corporations would be using many of 
them. However, this is not what we found. 


Half of the audit committees were 
operating below an effective level 


18.85 Half of the audit committees we 
examined were considered ineffective or 
only marginally effective. Of the 

14 committees, we found that: 


* only one followed most of the best 
practices and was performing effectively; 


* six followed many of the best 
practices and were reasonably effective; 


* five others used only some of the 
best practices and were only marginally 
effective; and 


An audit committee 
that operates 
effectively can bring 
significant benefits to 
the corporation. 


Half of the audit 
committees we 
examined were 
considered ineffective 
or only marginally 
effective. 
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Exhibit 18.2 


Selected Best Practices 
for Audit Committees 


Audit Committee Responsibilities 
The audit committee should ensure financial oversight by: 


* critically reviewing the interim and annual financial statements, the auditor’s report and the 
management discussion and analysis section of the annual report; and 


* actively soliciting the external auditor’s judgments about not only the acceptability but the quality 
of the corporation’s accounting principles as applied in its financial reporting. This discussion 
should include such issues as the clarity of financial disclosure and the aggressiveness or 
conservatism of the corporation’s accounting principles and estimates. 


The audit committee should ensure oversight of corporate books, records, financial and management 
control and information systems, and management practices by: 


* reviewing the special examination plan and report prepared by the external examiner; 


* actively soliciting information about significant risks and exposures and reviewing the adequacy 
of internal controls to manage those risks; 


° reviewing the integrity and effectiveness of the management information systems; 
e reviewing internal audit plans and reports and management’s subsequent actions; and 


* reviewing significant findings and recommendations made by the external auditor and examiner 
and following up on management’s subsequent actions. 


The audit committee should: 


° ensure ethical oversight through the annual review of management’s compliance with the 
corporate code of conduct; and 


* actively solicit all sensitive information (for example, senior management expenses, significant 
litigation, non-compliance with laws and regulations, misuse of corporate assets, illegal 
activities). 


Membership and Competencies 


The audit committee should be composed of at least three directors, the majority of whom should not 
be officers or employees of the corporation. 


Although a variety of skills and experience is beneficial to an effective and balanced audit 
committee, all members should be financially literate and at least one member should have 
accounting or related financial management expertise. Financial “literacy” signifies the ability to 
read and understand fundamental financial statements, including a balance sheet, income statement 
and cash flow statement, and the ability to ask probing questions about the corporation’s financial 
risks and accounting. “Expertise” signifies past employment experience in finance or accounting, 
requisite professional certification in accounting, or any other comparable experience or background 
that results in the individual’s financial sophistication (experience as a chief executive officer (CEO), 
for example, or other senior officer with financial oversight responsibilities). 


Operating Procedures 


Terms of reference. Audit committees should have clear, written terms of reference and operating 
procedures that specify the scope of the committee’s responsibilities and how it carries them out, 
including its structure, processes, and membership requirements. 


Meetings. The frequency of audit committee meetings should be tailored to the responsibilities 
assigned, but should be at least quarterly. The audit committee should also meet periodically with 
management, the external auditor and the head of internal audit, in separate private sessions. 
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* two were ineffective, despite using 
some of the best practices. 


The results demonstrate that the more best 
practices an audit committee uses, the 
more effective it is likely to be. 


Key gaps in practices 


18.86 Financial literacy and 
accounting expertise. The competencies 
and experiences of its members are an 
important factor in the audit committee’s 
effectiveness. Our assessment raised some 
important concerns: 


¢ There were three audit committees 
with at least one member who was not 
financially literate, and on one audit 
committee most members were not 
financially literate. Best practice 
recommends all members have the ability 
to read and understand fundamental 
financial statements and the ability to ask 
probing questions about the corporation’s 
financial risks and accounting. 


+ Two audit committees had no 
members with any accounting or financial 
management expertise. Best practice 
recommends at least one member have 
professional certification in accounting, or 
experience that leads to financial 
sophistication — employment as a CEO 
or senior financial officer, for example. 


18.87 Financial oversight 
responsibilities. A review of committee 
minutes indicated that audit committees 
generally follow only some of the best 
practices for ensuring financial oversight. 
Only two audit committees were 
following most of the best practices, 
including performing a critical review of 
interim and annual financial statements 
and the management discussion and 
analysis section of the annual report, 
soliciting the external auditor’s views on 
the quality of the corporation’s accounting 
principles, reviewing the external 
auditor’s significant findings and 
recommendations, and following up on 
corrective action by management. 
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18.88 | Crown corporation audit 
committees generally review internal 
audit plans and reports and the action 
taken in response to them by management. 
However, only three audit committees we 
looked at considered the integrity and 
effectiveness of management information 
systems, through brief discussion of 
corporate performance indicators. 


18.89 Operating procedures. The 
effectiveness of its communications with 
management and with the internal and 
external auditors is a factor in the 
effectiveness of the audit committee itself. 
Best practices include periodic private 
meetings with the auditors to give the 
audit committee a degree of independence 
from management and a further 
opportunity to freely ask probing 
questions. Only four of the audit 
committees were holding private sessions 
with the external auditor, and these varied 
in frequency. 


18.90 The audit committee of each 
Crown corporation should assess the 
extent to which it follows recognized 
best practices in order to identify areas 
that need improvement, and should take 
appropriate corrective action as 
necessary. It should seek input to this 
assessment from the internal audit 
group and the external auditor. 


The Crown corporation 
has a key role in 
interpreting its 
mandate. 


Approving Corporate Plans and 
Ensuring Mandate Relevance 


The corporate plan is the cornerstone of 
the control and accountability 
framework 


18.91 The Crown corporation has a key 
role in interpreting the mandate set out in 
its enabling legislation. The board of 
directors oversees the determination of the 
corporation’s core business and activities, 
its objectives for a five-year period and its 
strategy for achieving them, and the 
indicators and targets it will use to 
measure success. It must determine how it 
will balance its commercial objectives 
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The deficiencies in 
corporate plans 
undermine Crown 
corporation 
accountability. 


with its public policy objectives, and the 
trade-offs required to achieve that balance. 
For example, a corporation may need to 
strike a balance between services that are 
needed but will not produce profit and 
services that are targeted more 
commercially and generate profit. The 
corporation also needs to consider how it 
can contribute to government priorities 
and initiatives while still ensuring that its 
activities are consistent with its mandate. 
It must address all of these issues in a 
corporate plan that the FAA requires it to 
produce. Government approval of that 
plan is required for Crown corporations 
under Part X of the FAA. The corporate 
plan is the cornerstone of the control and 
accountability framework for Crown 
corporations. Where the government 
wishes to provide specific direction to a 
corporation, it may do so by directive 
powers under the FAA, subject to certain 
statutory limitations. 


18.92 The government is required to 
review the corporate plan before 
approving it in order to ensure that each 
Crown corporation’s strategy maximizes 
the achievement of its mandate. We 
expected that the government’s review 
and challenge would be rigorous, asking 
questions like the following: 


* Has the corporation properly 
interpreted its mandate? 


* Are the corporation’s objectives, 
strategies and targets appropriate and do 
its performance indicators provide a 
strong basis for holding it to account? 


- Are the trade-offs the corporation has 
made between its commercial objectives 
and its public policy objectives 
reasonable? 


* Do its performance targets 
sufficiently “stretch” the corporation? 


* Has the plan taken government 
priorities into account? 


* Is the corporation capitalized 
appropriately, and are targets for dividends 
and return on equity appropriate? 


« Has the corporation met its past 
performance targets? 


¢ Is there a need to assess whether the 
corporation’s mandate is still relevant? 


Many deficient corporate plans are 
approved, and the government has 
limited capacity to challenge them 


18.93 Our special examinations have 
found that there are significant 
deficiencies in some 38 percent of 
government-approved corporate plans, and 
less serious problems in a further 

28 percent. Problems include any or all of 
the following: 


* absence of long-term plans; 


* unclear or non-existent corporate 
objectives, targets, goals and business 
strategies, as well as weak action plans; 
and 


¢ little information by which to judge 
whether the corporation is achieving its 
objectives. 


18.94 These weaknesses undermine the 
corporate plan as a basis for ensuring that 
performance and accountability are 
maximized and the government’s 
objectives taken into account. They also 
indicate that the government’s process for 
approving corporate plans is deficient in 
challenging Crown corporations to 
achieve optimal performance. 


18.95 Crown corporation chairs and 
CEOs advised us that the government 
gave them little or no feedback on their 
corporate plans. They commented, “It was 
not always clear what the shareholder 
expected from its Crown corporations....It 
was too often a one-way street, with 
Crown corporations always feeding the 
system about what they were doing, but 
not hearing back from the shareholder 
about what they ought to be doing.” The 
Public Policy Forum’s 1998 report echoed 


Report of the Auditor General of Canada — December 2000 


that conclusion, noting, “There is little 
discussion about these corporate plans, 
and the plans themselves become more 
ritualistic than real in addressing the 
challenges and choices facing Crown 
corporations.” 


18.96 To determine why these problems 
exist, we looked at whether the 
government (responsible minister, 
Treasury Board and Department of 
Finance) has the capacity for rigorous 
review and challenge of corporate plans as 
a basis for their approval. 


18.97 Minister’s review. The 
responsible minister provides the first 
government review of a corporate plan. 
We found that many corporations receive 
little or no feedback on their corporate 
plan from their responsible minister. Yet 
Treasury Board guidelines say that the 
annual submission of the plan is meant to 
be a regular medium for the board and the 
responsible minister to clarify their 
respective appreciation of the 
corporation’s objectives. Generally, 
ministers have not responded formally to 
the corporate plan with an indication of 
areas of agreement as well as differences. 
The lack of formal communication is less 
troublesome when there is ongoing 
discussion between the minister and the 
Crown corporation. Half of the 
corporations we examined had a regular 
ongoing relationship with the minister. 
However, the other half saw their minister 
rarely, if ever, even when they requested a 
meeting and even when there were 
long-standing, unresolved issues. 


18.98 Crown corporations themselves 
could do more to be aware of the 
government’s concerns and priorities. For 
example, they could strengthen contacts 
with deputy ministers, ministerial staff or 
other officials to seek a better 
understanding of the government’s 
objectives. 


18.99 The government could also do 
more. The government itself has 
acknowledged the need to improve 
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communication. Government policy 
positions developed in 1982 stated: 


To improve communication, the 
government will institutionalize the 
concept of an annual shareholder’s 
meeting. Annually, each Minister as 
trustee owner, will meet with boards 
of directors to formally review 
corporate performance and to 
communicate specific government 
objectives to the board. This will not 
impede informal communication 
between the appropriate Minister and 
the corporation, but will underline the 
Minister’s responsibility to ensure 
that the chairman and board of 
directors of the corporation are kept 
fully informed of government 
objectives and that the board’s 
accountability to the government is 
maintained (Treasury Board of 
Canada, Policy Statement, 

30 June 1982). 


18.100 To properly exercise his or her 
responsibilities for a Crown corporation, 
the responsible minister must be able to 
obtain and rely on the related 
department’s advice. Government policy 
positions in 1982 also stated: 


To enhance its shareholder capability 
the government will improve or 
establish, where appropriate, 
machinery to assist Ministers with 
primary responsibility for wholly 
owned corporations. What is needed 
is...the establishment of effective 
support for Ministers...to provide 
needed direction, control and 
accountability....The government will 
seek to ensure that appropriate 
procedures and mechanisms exist to 
co-ordinate review and approval of 
corporate submissions, in order that 
the government might respond to such 
submissions in a thorough and timely 
manner....Specialized units have 
been, or now will be, established to 
assist Ministers in the performance of 
trustee shareholder or proprietor 
duties. Such units will have a direct 
reporting link to the Minister and his 
or her Deputy Minister. These 
arrangements will be developed to 
ensure that effective communication 
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Review by the 
Treasury Board and 
the Department of 
Finance is key to 
approving corporate 
plans. 


Exhibit 18.3 


Good Practices From Portfolio 
Management Groups 


between appropriate Ministers and 
corporate boards is maintained or 
enhanced (Treasury Board of Canada, 
Policy Statement, 30 June 1982). 


18.101 We found that many departments 
lack the capacity to support ministers in 
their responsibilities for Crown 
corporations. Policy groups in 
departments have played only a limited 
role, for various reasons. In some cases, 
the activities of the department and those 
of the corporation share little or no 
common ground. Some Crown 
corporations reported that departments 
lack knowledge about the commercial 
dimension of Crown corporations. 


18.102 Portfolio management represents 
the government’s most recent effort to 
increase departmental capacity in this 
area. The minister is asked to ensure that 
all organizations under his or her authority 
work together to support the minister and 
the government’s policies and programs. 
The Privy Council Office’s Advice to 
Ministers on Portfolio Management 
stresses that portfolio management must 
continue to respect the arm’s-length 
relationship between a Crown corporation 
and its responsible minister, the 
corporation’s purpose as an instrument of 
public policy, and in some cases, the 
ability of the corporation to compete in 
the marketplace. 


18.103 We found that portfolio 
management groups have generally not 
been active in supporting the ministers’ 
challenge and review of corporate plans. 
Only one of the portfolio management 


groups has advised the minister on 
whether or not to approve corporate plans. 


18.104 Portfolio management groups 
need to be sensitive to and respect the 
independence and autonomy of the board 
of directors in overseeing the management 
of Crown corporations. Their comments 
on corporate plans need to be at a strategic 
level. 


18.105 At the same time, we saw some 
good practices that could be applied more 
broadly, as Exhibit 18.3 shows. 


18.106 Review by Treasury Board and 
Department of Finance. After reviewing 
the corporate plan, the minister 
recommends it to the Treasury Board for 
approval; the Department of Finance also 
reviews it, if necessary. The Treasury 
Board looks at the strategic direction of 
each Crown corporation and the financial 
decisions or recommendations proposed 
by the responsible minister. It also 
approves the capital budget, certain 
transactions, and for most corporations, 
the operating budget. The Minister of 
Finance is responsible for the terms and 
conditions of borrowing plans and for 
directing that any surplus money held by a 
corporation be paid into the Consolidated 
Revenue Fund, with the concurrence of 
the responsible minister and the Governor 
in Council. 


18.107 We found that some Treasury 
Board Secretariat analysts have 
challenged Crown corporations to set 
more demanding objectives and targets, 
and this has produced better plans. 
Generally, however, Crown corporations 


° Some have regular meetings of the deputy minister and chief executive officers (CEOs), and 
the minister and chairs or CEOs, to discuss common issues and objectives. 


* One minister wrote letters to a corporation about government priorities. 


* One portfolio management group reviewed draft corporate plans, gave comments to the 
corporations and advised the minister on whether to recommend approval of the corporate 
plans. This group is considering the possibility of introducing a written response to corporate 
plans from the minister and an appointment letter to newly appointed chairs, indicating the 


government’s priorities. 
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report that Treasury Board Secretariat 
analysts often pay too little attention to 
major strategic issues and too much to 
minor issues related to funding. Crown 
corporations also report that some 
Secretariat analysts have lacked an 
understanding of the corporation and the 
commercial environment in which it 
operates, and did not know how to read 
financial statements. We noted that a 
number of the analysts are at junior levels 
and many have had no previous 
experience with Crown corporations. 
Treasury Board officials advise that 
resource reductions due to Program 
Review, and a less focussed specialization 
in Crown corporations, may have 
contributed to this situation. While the 
Treasury Board and the Department of 
Finance have promoted a team approach, 
it is not clear which one is assessing the 
capital structure of the large financial 
corporations; neither undertakes that role 
systematically. In light of the structural 
changes within the Treasury Board 
Secretariat and Finance and the 
introduction of portfolio management, 
there is a need to ensure that all the 
government players understand their 
respective roles and responsibilities in 
reviewing and challenging corporate 
plans. 


18.108 Crown corporations should be 
more proactive in finding ways to obtain 
information on government priorities 
and the government’s response to their 
corporate plans. 


18.109 ‘To align expectations, each 
corporation and the responsible 
minister should reach an understanding 
on the most effective ways to outline 
priorities, provide feedback and reach 
consensus on corporate plan 
submissions and to maintain ongoing 
contact between them. 


18.110 The government should 
strengthen its capacity to ensure that its 
approval of corporate plans is based on 
a rigorous challenge and review. It 
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should strengthen departmental 
capacity to support ministers in their 
responsibilities for Crown corporations, 
and should ensure that all government 
players understand their respective 
roles and responsibilities in approving 
corporate plans. 


Need for a more systematic review of 
corporation mandates 


18.111 Periodically, there is a need to 
consider whether the mandate of a Crown 
corporation is still relevant. We found that 
in reviewing corporate plans, Treasury 
Board Secretariat analysts have sometimes 
commented on this. In other cases a 
change in government or minister, a 
significant change in the economy, or a 
Crown corporation’s request for a 
legislative change can prompt a review of 
mandate relevance. Boards of directors 
clearly have an interest in mandate 
relevance because the mandate affects the 
choice of strategic direction and the 
development of the corporate plan. In all 
cases, the mandate review is a 
fundamental assessment of the continuing 
relevance of a Crown corporation’s role. 
Such reviews generally go well beyond 
the scope of the annual corporate plan and 
are vital for ensuring that the corporation 
remains a cost-effective instrument of 
public policy. 


Periodically, there is a 
need to consider 
whether the mandate 
is still relevant. 


18.112 A decision to proceed with a 
mandate review needs to be based on such 
factors as an ongoing assessment of 
changes in the operating environment; an 
expressed need to make changes; a request 
by the corporation for a mandate review 
or for legislative amendments that affect 
the corporation’s mandate; or an 
unresolved conflict related to the mandate 
that is holding up the shareholder’s 
approval of the corporate plan. 


18.113 We expected to find that mandate 
reviews are carried out regularly, reported 
to all parties in the accountability 
framework, and engage all parties 
concerned, including Parliament. What we 
found however, is that only two 
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corporations are required by their 
respective enabling legislation to undergo 
periodic mandate reviews. 


18.114 We found that mandate reviews 
have been carried out by a corporation or 
the responsible minister’s department on a 
generally ad hoc basis. Those done by the 
corporation were not always brought to 
the attention of the Treasury Board, the 
minister, or Parliament; nor did they 
engage all of the parties concerned, 
including Parliament, or address all of the 
relevant issues. Mandate reviews led by 
the minister, and supported by outside 
expertise as necessary, have a better 
record of engaging all parties and bringing 
out issues that may not be identified in a 
review conducted by the corporation 
itself. We note that a legislative provision 
for mandate review ensured that it was 
carried out when otherwise it might not 
have been. 


18.115 We noted that some 
parliamentary committees have held 
hearings on the relevance of Crown 
corporation mandates. In one case, a 
parliamentary committee had a major 
impact on the corporation’s move toward 
providing more service to a group that is a 
government priority. Generally, however, 
Parliament has little direct involvement 
with Crown corporations. It could take 
more opportunity to meet with Crown 
corporations and have them account for 
their results. 


18.116 The government could consider 
the following guidelines for conducting 
mandate reviews: 


* Each review assesses the current 
validity of the mandate, the continuing 
need for the corporation, and its record of 
cost-effective performance in meeting the 
needs implicit in the mandate. 


+ Mandate reviews are led by the 
minister and supported by outside 
expertise as necessary. 
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- The corporation provides input to the 
minister-led review, but does not conduct 
it or contract a third party to undertake it. 


¢ The minister, in consultation with the 
corporation and central agencies, 
determines the terms of reference for the 
review with a focus on the critical 
elements of the mandate. It may be 
appropriate to limit the scope of some 
reviews to particular aspects of the 
mandate. 


¢ Any third party contracted to 
conduct the review consults extensively 
with all parties concerned, inside and 
outside government. 


* The results of mandate reviews are 
shared with all parties in the 
accountability framework — corporate 
management, the board, Treasury Board, 
the Department of Finance, the 
responsible minister and Parliament. 


* Parliament conducts hearings on the 
results of mandate reviews, as appropriate. 


18.117 The government should 
systematically consider, at least every 
10 years, whether a review of each 
Crown corporation’s mandate is 
warranted. This should be triggered by 
regulation or some other mechanism. 
The government should develop 
guidelines for conducting mandate 
reviews, and should ensure that the 
results of the mandate reviews are 
transparent and reported to all parties 
in the accountability framework. 


Governance Protocol Between 
Ministers and Crown Corporations 


18.118 In previous sections of this 
chapter we have cited the need for 
dialogue to reach an understanding 
between each Crown corporation and the 
responsible minister on the 
implementation of key aspects of 
governance, including: 


* how the board of directors will be 
involved in the appointment of the board, 
the chair and the CEO; 
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- how the government will 
communicate any issues it wants a Crown 
corporation to take into account in its 
corporate plan; 


« mechanisms for ministerial feedback 
on the corporate plan; and 


* procedures for handling ad hoc 
issues and maintaining ongoing contact 
between ministers and corporations. 


18.119 In addition, the dialogue could 
cover elements such as: 


* expectations for performance 
monitoring; 


* performance evaluations of boards 
and CEOs and their remuneration; and 


* potential conflicts of interest. 


18.120 Each minister’s situation is 
unique, and strategies need to reflect the 
minister’s needs and demands as well as 
the needs of the corporation. Each time 
there is a change of minister or a new 
chair is to be appointed, the agreement 
would be reconsidered and reviewed. The 
key to success for both parties would be 
managing expectations. As a means of 
confirming the agreement, the chair of the 
corporation could record his or her 
understanding of the agreement in writing 
and the minister could outline the 
agreement in a letter upon the chair’s 
appointment. The agreement thus 
documented would constitute a 
governance protocol. Prospective new 
board members would be given a copy of 
the protocol. A protocol that gave the 
board a meaningful role would help attract 
strong board members, and this would 
serve the interests of both the corporation 
and the government. 


18.121 Each Crown corporation and 
the responsible minister should reach a 
common understanding on the 
implementation of key aspects of 
governance, and they should record that 
understanding. 
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Conclusion 


18.122 The management of Crown 
corporations has improved in the 15 years 
since the Financial Administration Act 
was amended to introduce a strengthened 
Crown corporation accountability regime. 
The results of our special examinations 
bear this out. Nevertheless, further 
improvement is needed in some important 
areas of Crown corporation management 
like strategic and corporate planning and 
the measurement and reporting of 
performance. 


18.123 Our findings on the process for 
appointing boards of directors converge 
with those of other studies to confirm that 
there is a need for the federal government 
to revisit and strengthen the process. 
Boards of directors are responsible for the 
affairs of the corporation. Yet boards lack 
essential skills and capabilities that are 
required to effectively carry out their role. 
The government needs to meaningfully 
involve boards in their renewal. Boards 
and the government need to outline their 
respective skill and capability 
requirements, and the government needs 
to act on them. Boards of directors also 
need to take a more active role in their 
chairs’ selection. 


A governance protocol 
that gave the board a 
meaningful role would 
help attract strong 
board members. 


18.124 The government needs to 
improve the timeliness of its appointment 
decisions and maintain its progress in 
extending the average length of time 
served by qualified directors. This will 
ensure a nucleus of seasoned directors 
who can provide institutional knowledge 
and experience. 


18.125 We found that accountability of 
the chief executive officer to the board is 
best reinforced when the board takes the 
lead role in the CEO’s selection, with the 
government providing input and making 
the final decision. A transition strategy 
may be needed to reach this goal for some 
boards that do not yet have the capacity to 
lead the selection process. 
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18.126 The effectiveness of audit 
committees is assuming ever-greater 
importance. We found that the practices of 
audit committees need to be strengthened 
if they are to more effectively discharge 
their key roles and responsibilities in 
Crown corporation governance and the 
financial reporting process. A 
strengthened audit committee will mean a 
stronger board of directors. 


18.127 The corporate plan is the 
cornerstone of the control and 
accountability framework for Crown 
corporations. We found deficiencies in 
many government-approved corporate 
plans, and a limited capacity in 
government to review and challenge those 
plans as a basis for their approval. These 
weaknesses undermine the corporate plan 
as a basis for ensuring that performance 
and accountability are optimized and that 
the government’s priorities have been 
addressed. 


18.128 Periodically, there is a need to 
verify whether the mandate of a Crown 
corporation is still relevant. Mandate 
reviews are the standard tool for this. 
They have proved useful in realigning 
Crown corporation mandates. We found, 
however, that the need for mandate 
reviews needs to be questioned more 
regularly, the results more widely 
reported, and the reviews led by the 
government, with the involvement of 
Parliament. 


18.129 There is a need for dialogue and 
consensus between each Crown 
corporation and the responsible minister 
on how each party will be engaged in 
implementing key aspects of governance. 
The board of directors of each Crown 
corporation and the responsible minister 
need to document their shared 
understanding of how these key aspects of 
governance will be implemented, thereby 
creating a governance protocol. 


18.130 We plan to follow up within 
two years on the recommendations we 
have made in this chapter, including those 


on the quality and composition of boards 
of directors. We will also review the 
progress made on the recommendations as 
we conduct our annual audits and special 
examinations in individual Crown 
corporations. 


Privy Council Office’s response: Good 
corporate governance is essential if a 
Crown corporation is to fulfil its mandate. 
The government has made strides in a 
number of corporate governance areas, 
including improving reappointment 
statistics (64 percent of incumbents 
eligible for reappointment are reappointed 
as directors for a second or third term) 
and ensuring that appointments reflect 
Canada’s diversity. 


Improved and increased communication 
among all the interested partners will be a 
critical factor in addressing concerns 
identified in the study and in responding 
to the recommendations put forward. 
Measures will be examined to address 
areas needing improvement and to 
ultimately strengthen corporate 
governance in Crown corporations. 


It is paramount that appointments to 
Crown corporations result in strong 
boards of directors and that the 
appointment process for Crown 
corporations is inclusive of all interested 
parties, bearing in mind that appointments 
to boards of directors of Crown 
corporations, including those of the chair 
and the chief executive officer, remain at 
the discretion of the government. 


Crown corporations’ responses: It is not 
practical to obtain the responses of 

41 Crown corporations to the 
recommendations that apply to them 
directly. However, we did discuss our 
findings and recommendations with the 
chairs and CEOs of 10 Crown 
corporations. Generally, they supported 
the main messages of the chapter and, for 
the most part, agreed with the 
recommendations directed specifically at 
Crown corporations. 
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ty About the Audit 


Objectives 


The audit sought to assess how well three key governance features of the Financial Administration Act (FAA) 
are functioning in Crown corporations. These include the appointment process for directors, CEOs and chairs; 
the capacity of the government to review and challenge corporate plans and ensure mandate relevance; and 
the composition, roles and responsibilities and performance of audit committees. 


In addition, we wanted to provide a summary of the results of the three cycles of special examinations in 
Crown corporations. 


Scope 


The audit scope included a review of the key responsibility and accountability provisions of Part X of the FAA 
and parallel provisions in the enabling legislation of Crown corporations exempted from Part X. We examined 
the practices of these Crown corporations and the various shareholder players to support the process for 
appointing directors, chief executive officers (CEOs) and chairs, to review and challenge corporate plans, and 
to ensure mandate relevance. 


We compared these practices with standards and practices of state-owned enterprises in industrialized 
countries. In addition, we interviewed chairs and CEOs, and examined appointment data from 15 diverse 
Crown corporations. We also examined the composition, roles and performance of audit committees in the 
same sample. We further examined recent chair, CEO and director appointments in eight corporations. We 
used the results of other research, studies, audits and roundtables to provide supporting evidence for our 
findings. All recommendations were made within the parameters of the existing FAA accountability and 
control framework, and respected the Governor-in-Council authority for appointments. 


The chapter also provides a summary of the results of the third cycle of special examinations. Special 
examinations are a form of value-for-money audit carried out in Crown corporations. 


Criteria 


The following general audit criteria were applied. We expected that in order to protect the shareholder’s 
interest, to ensure that Crown corporations achieve their public policy objectives effectively and efficiently, 
and to optimize their commercial objectives: 


¢ There should be timely appointments of qualified board chairs, CEOs and board directors that meet the 
requirements of the government and the corporation and strengthen the accountability relationships 
among the board, the CEO and the government. 


¢ Audit committees should be constituted with appropriate skills and experience, carry out necessary roles 
and responsibilities, and perform their duties effectively, as measured against best practices. 


¢ The government should conduct a robust review and challenge of a corporation’s corporate plan, as the 
basis for the plan’s approval. 


* There should be assurance that the mandate of each corporation, including its public policy and financial 
objectives, continues to be relevant. 
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Appendix A 


List of Crown Corporations by Responsible Minister 


ES Minister Responsible for | Crown Corporation 


Agriculture and Agri-Food Canada Canadian Dairy Commission 
| Farm Credit Corporation 


Canadian Heritage Canada Council* 
Canadian Broadcasting Corporation* 
| Canadian Film Development Corporation* 
Canadian Museum of Civilisation 
Canadian Museum of Nature 
Canadian Race Relations Foundation* 
| National Arts Centre Corporation* 
National Capital Commission 
National Gallery of Canada 
National Museum of Science and Technology Corporation 


Department of Finance Bank of Canada* 

Canada Deposit Insurance Corporation 
Canada Development Investment Corporation 
CPP Investment Board* 

Petro-Canada Limited 


Fisheries and Oceans Freshwater Fish Marketing Corporation 
Foreign Affairs International Development Research Centre* 
International Trade Canadian Commercial Corporation 


Export Development Corporation 


Industry Canada Business Development Bank of Canada 
Enterprise Cape Breton Corporation 
Standards Council of Canada 


“Natural Resources Canada Atomic Energy of Canada Limited 
Cape Breton Development Corporation 


Public Works and Government Services Canada Canada Lands Company Limited 

Canada Mortgage and Housing Corporation 
| Canada Post Corporation 

Defence Construction (1951) Limited 
Queens Quay West Land Corporation 
Royal Canadian Mint 


Atlantic Pilotage Authority 

Federal Bridge Corporation Ltd., The 
Great Lakes Pilotage Authority 
Laurentian Pilotage Authority 
Marine Atlantic Inc. 

Pacific Pilotage Authority 

Ridley Terminals Inc. 


Transport Canada 


Treasury Board 
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Appendix B 


Roles and Responsibilities Under the Financial Administration Act 


Crown Corporation G Shareholder 
Treasury 
Chief Board, Finance 
Executive and Privy 
Officer (CEO)/ Responsible Council Office | Governor in 
Management | Board Minister (PCO) Council Parliament 
| Creation, Acquisition, Disposal and Dissolution 
Parents Recommends | Review Approves 
Subsidiaries Recommends | Review Approves 
Appointments E a 
CEO and Provides Consults Appoints 
Chair advice 
Other Appoints Approves 
Directors 
Officers of Recommends _ | Appoints 
Corporation 
Remuneration and Other Benefits for Chairs, CEOs and Directors 
Remuneration Approves Approves the 
and Benefits benefits other rate of 
than remuneration 
remuneration and CEO 
and performance 
recommends pay 
CEO 
performance 
pay 
Directives and Regulations 
| Directives Recommends Approves Receives 
and tables 
copy with 
Parliament 
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Appendix C 


General Information on Special Examinations and Overview of Results 


What are special examinations? 


A special examination is a type of value-for-money audit of a Crown corporation. It serves to provide an independent 
opinion to the board of directors on whether the corporation’s financial and management control and information 
systems and management practices have been maintained so as to provide reasonable assurance that: 


® the assets of the corporation have been safeguarded and controlled; 


e the financial, human and physical resources of the corporation have been managed economically and efficiently; 
and 


¢ the operations of the corporation have been carried out effectively. 


Examining all systems and practices in detail is impractical and costly. Therefore, based on a thorough understanding 
of the corporation and its operating environment, and on a risk analysis, the examination concentrates on areas of the 
corporation where any deficiencies that might exist could be significant. 


All parent Crown corporations named in Schedule III of Part X of the Financial Administration Act (FAA), and their 
wholly owned subsidiaries, are subject to special examinations. Crown corporations undergo a special examination 
every five years. The first five-year cycle of special examinations was conducted between 1984 and 1990, and the 
second between 1990 and 1996. The third cycle of examinations commenced in 1996 and will be completed by 2001. 


How many Crown corporations are subject to special examinations? 


Thirty-nine Crown corporations were subject to a special examination during the third cycle. The Office has carried 
out 29 of 31 examinations to date (two as joint examiner), while private sector practitioners have conducted 8. The 
results discussed hereunder are based only on the examinations carried out by the Office of the Auditor General. 


Our special examination findings indicate that management of Crown corporations is improving. Nevertheless, 
there are important areas that continue to require further improvement. 


Types of opinions reported 


The following table sets out the types of opinions reported in all three cycles of special examinations (SEs). A “clean 
opinion” indicates that no significant deficiencies were reported. “Significant deficiencies” indicate one or more 
instances where there wasn’t reasonable assurance that assets were being safeguarded and controlled, resources 
managed efficiently and economically, or operations carried out effectively. 


| 15t Cycle - 2d Cycle - 34 Cycle - 
| Type of Opinion 26 SEs 32 SEs 29 SEs 
Clean opinion 23% 22% 52% 
‘Significant deficiencies 77% 78% 48% 


When comparing results of the third cycle to those of previous cycles, we see a significant improvement in the 
proportion of special examinations that had clean opinions. 


To whom were special examinations reported? 


When a special examination contains information that, in the examiner’s opinion, should be brought to the attention of 
the appropriate minister or Parliament, the examiner may do so. (This provision of the FAA does not apply to Crown 
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corporations named in Part 2 of Schedule III of the Act and most exempt Crown corporations.) Reporting beyond the 
board of directors is usually done when the significant deficiency is beyond the corporation’s ability to address on its 
own, or when significant deficiencies noted in a previous special examination have not been adequately addressed, or 
when several significant deficiencies have been identified in a corporation’s systems and practices. Otherwise, the 
report is provided to the board of directors only. The following table shows the proportions of special examinations 
reported to the board and beyond. 


E Special examinations reported to 15 Cycle 24 Cycle | 3°4 Cycle 
Board of directors only 81% 63% 76% 
“Minister 11% 37% | 24% 
Parliament 8% - - 


The reduction in the number of reports brought to the attention of the appropriate minister in the third cycle is directly 
related to the reduction in the number with significant deficiencies; four reports with significant deficiencies that went 
to the appropriate minister in the second cycle, had clean opinions in the third cycle. The fact remains, however, that 
one quarter of Crown corporations had significant deficiencies that the examiner felt should be brought to the attention 
of the appropriate minister. 


What significant deficiencies were reported? 


The following tables identify the systems and practices where significant deficiencies were found (one significant 
deficiency may affect more than one system). While only significant deficiencies were identified in first and second 
cycle special examination reports, third cycle reports also included other important deficiencies that the examiner 
thought should be reported to the board. For the third cycle, the table identifies the systems and practices where 
significant deficiencies were found, and gives an additional column combining them with other deficiencies found in 
the third cycle. 


Percentage of 
Crown corpora- 
tions with both 
significant and 
Percentage of Crown corporations with other 
Systems and Practices (Areas) significant deficiencies in identified areas deficiencies 
15' Cycle 24 Cycle 3'4 Cycle 3™4 Cycle 
Corporate and strategic planning 50 aie) 38 66 
‘Performance measurement and reporting 46 eh 24 | 76 
Operations 65 44 17 66 
Human resources management 42 28 10 62 
“Asset and facilities management 46 25 7 i 14 
‘Risk management ; 15 2 7 14 
‘Environment 8 9 7 21 
Marketing | 27 [2 3 31 ; 
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Percentage of Crown corporations with significant deficiencies in 
corporate and strategic planning and/or performance measurement and reporting 
Includes other 
deficiencies 
15t Cycle 2"4 Cycle 3" Cycle 3™4 Cycle 
62 78 38 90 


The reduction in the systems and practices with significant deficiencies in the third cycle is directly related to the 
overall reduction in the number of special examinations with significant deficiencies. However, it is still of concern 
that 38 percent of Crown corporations had significant deficiencies related to corporate and strategic planning and 
performance measurement and reporting (90 percent had significant and other deficiencies combined). This indicates 
that these corporations do not: 


¢ have well-articulated corporate visions, values and goals; 
¢ have clearly expressed strategies or action plans to achieve their mission and vision; 
¢ know the extent to which they are achieving their objectives; or 


* report adequately on these objectives and strategies to Parliament and government. 
Environmental and sustainable development issues in Crown corporations 


Of the 29 special examinations of Crown corporations completed so far in the third cycle, 20 included the management 
of environment and sustainable development issues. Those issues were excluded from the examinations of the 9 other 
corporations after our survey found no potentially significant environmental risks. Two of the 20 special examinations 
reported significant deficiencies related to the corporations’ systems and practices for the environment and sustainable 
development. Neither corporation had an environmental management system (a formal environmental policy, 
environmental objectives, measurable targets, and a process to ensure that environmental and sustainable development 
considerations are taken into account in decision making). 


Some other areas for improvement were identified. These included the need for: 


* amore comprehensive environmental management system (for example, a corporation might need to improve 


information, or to formalize and document existing procedures); 
improved measurement and reporting of environmental performance; 
more training in environmental management; and 


a policy on environmental considerations in the awarding of offshore contracts. 


18-36 Report of the Auditor General of Canada — December 2000 


Governance of Crown Corporations 


Appendix D 


Best Practices Applicable to Federal Crown Corporation Audit Committees 


In this appendix, we present a comprehensive list of suggested best practices that are applicable to federal Crown 
corporations and are complementary to the statutory requirements of audit committees under Part X of the Financial 
Administration Act. These best practices were taken from recent studies and publications in Canada and the U.S., 
including the Report and Recommendations of the Blue Ribbon Committee on Improving the Effectiveness of 
Corporate Audit Committees, sponsored by the New York Stock Exchange and the National Association of Securities 
Dealers (1999), as well as various accounting and auditing publications. 


We do not suggest that this template of best practices applies equally to all audit committees. Each audit committee 
should develop and observe guidelines suited to itself and its corporation. Some of these responsibilities may also be 
assigned to other board committees. However, as part of the audit committee’s periodic self-assessment, it should 
ensure that key oversight responsibilities outlined in the best practices are assumed by it or another board committee. 


Audit Committee Best Practices 


(Those in bold were selected for comparison with current Crown corporation practices, as described in the chapter.) 
Audit Committee Responsibilities 


The audit committee should: 


* ensure oversight of compliance matters by monitoring the corporation’s compliance with applicable laws and 
regulations. 


* ensure financial oversight by: 


- critically reviewing the operational and capital budgets, interim and annual financial statements, the 
auditor’s report and the management discussion and analysis section of the annual report; and 


- reviewing the external auditor’s audit plans and actively soliciting his/her judgments about the quality, not 
just the acceptability, of the corporation’s accounting principles as applied in its financial reporting. 
This discussion should include such issues as the clarity of financial disclosures and the degree of 
aggressiveness or conservatism of the corporation’s accounting principles and estimates. 


* ensure oversight of corporate books, records, financial and management control and information systems 
and management practices by: 


- reviewing the special examination plan and report prepared by the external examiner; 


- actively soliciting information about significant risks and exposures and reviewing the adequacy of 
internal controls to manage those risks; 


- reviewing the integrity and effectiveness of the management information systems; 
- reviewing internal audit plans and reports and management’s subsequent actions; and 


- reviewing significant findings and recommendations made by the external auditor and follow up on 
management’s subsequent actions. 
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® confirm the external auditor’s independence through the receipt of a formal written statement from the external 
auditor and through subsequent dialogue about any issues that may impact the objectivity and independence of the 
auditor. 


* ensure ethical oversight through the annual review of management’s compliance with the corporate code of 
conduct. 


° actively solicit all sensitive information (for example, senior management expenses, significant litigation, 
non-compliance with laws and regulations, misuse of corporate assets, illegal activities). 


The audit committee should, where warranted, advise the board of directors about any of these matters. 
Membership and Competencies 


The audit committee should be composed of at least three directors, the majority of whom should not be officers 
or employees of the corporation. Audit committee members must have a significant degree of commitment to the 
corporation that they take adequate time for meeting preparation, near-perfect meeting attendance, and ongoing 
education about the corporation’s business and environment and topical issues. 


Good governance dictates that the board be composed of individuals with certain personal characteristics such as 
integrity, strategic thinking, and the ability to ask probing questions. Ideally, audit committee members would be the 
most qualified and experienced directors on the board. 


Although a variety of skills and experience is beneficial to an effective and balanced audit committee, all 
members should be financially literate and at least one member should have accounting or related financial 
management expertise. Financial “literacy” signifies the ability to read and understand fundamental financial 
statements, including a balance sheet, income statement and cash flow statement, and the ability to ask probing 
questions about the corporation’s financial risks and accounting. “Expertise” signifies past employment 
experience in finance or accounting, requisite professional certification in accounting, or any other comparable 
experience or background that results in the individual’s financial sophistication (such as CEO or other senior 
officer with financial oversight responsibilities). 


Operating Procedures 


Terms of Reference. Audit Committees should have clear, written terms of reference and operating procedures 
that specify the scope of the committee’s responsibilities, and how it carries out those responsibilities, including 
structure, processes, and membership requirements. This charter should be approved and reassessed periodically by 
the board of directors. 


Authority. The audit committee should have explicit authority to investigate any matters within its terms of reference, 
should be provided with the resources it needs, and should have full access to information and the organization’s 
personnel. The committee should also be able to obtain independent professional advice. 


Member Orientation. The audit committee should consider training and education programs for its members to 
ensure that they have current knowledge of the following: 


* the committee’s responsibilities and the methods of discharging them; 
the roles of the internal and external auditors; 


the corporation’s business, including products, systems, risks and opportunities; and 


basic elements of technical areas such as accounting principles and policies, internal control systems, and 
auditing. 


Meetings. The frequency of audit committee meetings should be tailored to the responsibilities assigned, but 
should be at least quarterly. The audit committee agendas are set with sufficient input from the chair of the 
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committee, the CEO, the chief financial officer and the internal and external auditors. Detailed minutes of the 
meetings should be prepared. The external auditor should be invited to every meeting. The audit committee should 
also periodically meet with management, the external auditor and the director of internal auditing in separate 
private sessions. 


Reporting and Self-Assessment. The audit committee should periodically report the results of its activities to the 
board of directors. Its charter should be disclosed at least triennially in the corporation’s annual report. The committee 
should also provide a general disclosure letter in the corporation’s annual report stating whether the audit committee 
has satisfied its responsibilities during the prior year in compliance with its charter. The disclosure should also state 
that the audit committee: 


* has reviewed the financial statements with management and has discussed the quality of the accounting principles 
and significant judgements; 


* has discussed the audited financial statements, accounting principles and significant judgements with the external 
auditor; 


* has discussed the information provided by management and the external auditor; and 


¢ believes, in reliance on the review and discussions with management and the auditors, that the corporation’s 
financial statements are fairly presented in conformity with generally accepted accounting principles in all 
material respects. 


The audit committee should periodically assess its effectiveness and the adequacy of its mandate, and its chair should 
also periodically assess the performance of individual committee members. 
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Chapter 19 


Reporting Performance to Parliament 


Progress Too Slow 


The audit work reported in this chapter was conducted in accordance with the legislative 


mandate, policies and practices of the Office of the Auditor General. These policies and 
practices embrace the standards recommended by the Canadian Institute of Chartered 


Accountants. The numbered paragraphs in bold face represent recommendations. 


Table of Contents 


Main Points 


Introduction 


A good start in 1997 

Many jurisdictions report on performance 
The Expenditure Management System 
Clarifying results and performance concepts 
Focus of the audit 


Observations and Recommendations 


The Expenditure Management System Since 1997 
What We Expected in Reports 


Some Improvement in Stating Performance Expectations 
A variety of terms used to state what is to be accomplished 
Setting concrete expectations remains a challenge 
Inconsistent use of concepts 


Slow Progress in Reporting Accomplishments 
Improvements needed in reporting accomplishments 
More effort is needed to assess a program’s contribution 
Evaluation findings can play an important part in reporting performance but are underused 


Need to Improve Balance and Indicate Reliability of Data 
More balanced reporting needed to enhance credibility 
More attention is needed to data reliability 


Performance Reporting Shows Modest Improvement 
Some departments show progress in communicating performance 
Good performance reporting overall is still some years away 


Treasury Board Secretariat Leadership for Reporting 
Better electronic reporting needs to be encouraged 
Treasury Board Secretariat is focussing on collective reporting 


More Involvement of Parliamentarians Is Needed 
Use by parliamentarians still limited 
Improvements proposed to the current regime for reporting to Parliament 
Members’ suggestions for improvement 


Changes Are Required if Performance Reporting Is to Improve 
The reporting regime has strengths 
What is the problem? 
Clear and common reporting principles are needed 
Performance is reported in a political environment 
More consideration of incentives needed 
A stronger challenge of expectations is required 
A stronger role for parliamentary committees 
Is it time for legislation? 


Conclusion 


About the Audit 


Report of the Auditor General of Canada - December 2000 


Page 


19-5 


19-7 
19-7 
19-7 
19-7 
19-8 
19-8 


19-8 
19-§ 
19-11 


19-11 
19-11 
19-12 
19-13 


19-15 
19-15 
19-17 
19-18 


19-19 
19-19 
19-19 


19-21 
19-21 
19-22 
19-23 
19-24 
19-24 
19-25 
19-25 
19-25 
19-26 
19-27 
19-27 
19-27 
19-28 
19-28 
19-29 
19-30 
19-3] 
19-3] 


19-33 


19-35 


19-3 


Page 


Exhibits 
1931 Terms and Concepts of Results and Performance 19-9 
1972 What Should Be in a Performance Report 19-11 
19.3 Assessing Statements of Expected Performance 19-13 
19.4 Statements of Performance Expectations — Examples 19-14 
1935 Assessing How Well Accomplishments Are Reported 19-15 
19.6 Types of Accomplishments Reported 19-16 
IOo7/ Reporting Activities as Accomplishments — Examples 19-17 
19.8 Using Evaluation Findings to Strengthen a Performance Story — Example 19-18 
19.9 Good Practices to Improve the Quality of Performance Information 19-20 
19.10 Accountability Legislation in Other Jurisdictions — Examples 19-32 
Appendices 
AS Departments and Agencies Whose Reports We Examined 19-37 
B. Setting Outcome Expectations 19-38 
C. Status of Our 1997 Recommendations on Performance Reporting 19-39 
D. Summary of Recommendations of the 1998 Standing Committee on Procedure and 

House Affairs, and the Government’s Response 19-4] 
EB; Summary of Proposals for Improved Reporting to Parliament Project (Phase 2) 

and Subcommittee Recommendations 19-42 


19-4 Report of the Auditor General of Canada - December 200! 


Reporting Performance to Parliament 


Progress Too Slow 


Main Points 


19.1 Federal departments and agencies have made some progress in reporting on their performance to 
Parliament, but we are disappointed at its present pace. Continued at this rate, it would take too long for good 
reporting to become routine. We recognize that reporting on performance is not easy, and it takes time. Still, we 
believe that the effort deserves more attention: members of Parliament need information on what Canadians are 
getting for their taxes in order to hold the government to account. Good performance reporting has become an 
essential part of modern democratic systems. 


19.2 To that end, it may be time for legislation. Several jurisdictions in Canada now have legislation requiring 
government to report on its performance to the legislature. Legislation highlights the importance of good reporting 
and puts it on a more permanent and stable footing. 


19.3 Our examination of the Estimates documents — reports on plans and priorities and departmental 
performance reports — of 47 departments and agencies over the past four to five years found the following: 


* While there has been progress, few departments set out concrete statements of what they expect to 
achieve and then report back specifically against those expectations. 


* In reporting what they have accomplished, departments focus too much on listing activities and 
outputs and too little on linking them to and reporting on the intended outcomes. 


¢ Reporting makes too little use of evaluation findings on the results that activities are accomplishing 
and on how programs are contributing to outcomes. 


¢ There is little linking of financial and non-financial performance information; most accomplishments 
are not costed. 


* Performance reports lack balance. Most departments report only the “good news” and make little 
mention of performance that did not meet expectations. As members of Parliament have said 
repeatedly, reports that present only good news are not credible. 


19.4 We noted that the Treasury Board Secretariat has provided leadership for reporting. And we did find 
pockets of good reporting and a number of strengths in the present regime of reporting. Nevertheless, after five 
years of experimenting, departmental reporting overall has only improved modestly. 


29.5 Three factors stand out as contributing to the current weak state of reporting: 
* basic principles of good reporting are not understood or applied; 
* performance reporting has political dimensions; and 
* there are few incentives for good reporting or sanctions for poor reporting. 


19.6 In addition to considering legislation, in our view the government needs to make the following changes 
in the present reporting regime: 


* reach agreement on principles for reporting performance information; 
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* give more attention to the right incentives for incorporating evidence-based approaches into 
reporting; 


* ensure stronger oversight by the Treasury Board Secretariat; and 


* seek to enhance the involvement of parliamentarians in reviewing performance plans and reports. 


Background and other observations 


19.7 Federal departments submit performance reports to Parliament in the fall every year, outlining what they 
have accomplished. The reports are subject to consideration by their respective parliamentary standing 
committees. Review of the reports at that time would enable parliamentary committees to make recommendations 
for the upcoming spending decisions to be announced by the government later, in the Budget. In the spring, after 
the Budget, departments submit reports on their plans and priorities, describing what they are setting out to 
achieve in the upcoming year and beyond. 


19.8 In 1997 we first reported on this new regime for reporting to Parliament. We found that a good start had 
been made and that the basic framework used for reporting was sound. 


19.9 Credible reporting on performance has a number of benefits: it can enhance accountability and trust in 
government, provide information for improved programs and services and better parliamentary scrutiny of them, 
and serve as an incentive for departments and agencies to manage for the results they have set out to achieve. 


19.10 On two occasions in recent years, the Standing Committee on Procedure and House Affairs has issued 
reports with recommendations on ways to provide for greater involvement of parliamentarians in the review of the 
Estimates. 


19.11 The President of the Treasury Board annually tables, along with the departmental performance reports, a 
report in Parliament on government performance measurement, reporting and results-based management. In recent 
years it has provided useful performance information on collective results — areas where several departments or 
several governments are working collectively toward a result. 


19.12 The Treasury Board Secretariat has taken important steps toward more electronic reporting. It needs to 
encourage departments to make greater use of this vehicle, and gather and share good practices. 


The response of the Treasury Board Secretariat, on behalf of the government and the departments audited, 
is included at the end of this chapter. The Secretariat has responded positively to our chapter and 
recommendations therein. The Secretariat accepts our recommendations and indicates the actions that, 
where appropriate, it is taking or intends to take to address them. 
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A good start in 1997 


19.13. In 1997, we examined the state of 
the federal government’s performance 
reporting to Parliament (Auditor General’s 
1997 Report, Chapter 5, Reporting 
Performance in the Expenditure 
Management System). We noted that the 
public and members of Parliament often 
lacked the kind of information that would 
demonstrate what Canadians were getting 
for the taxes they paid. Too often, 
departments described the activities they 
had undertaken and the services they had 
delivered but not the results they had 
achieved. 


19.14 However, we did observe that 
several departments had made progress; 
we found aspects of good performance 
reporting in the information they had 
provided to Parliament. But we noted that 
in describing their expected performance, 
departments needed to be clearer and 
more concrete; in reporting their actual 
performance, they needed to focus more 
on the benefits they had gained for 
Canadians. The chapter made a number of 
recommendations for departments to 
improve their performance reports. 


19.15 We also recommended that the 
Treasury Board Secretariat strengthen its 
leadership role. We noted that the role of 
parliamentary committees was vital to 
continued progress: if committees asked 
for information on performance and 
visibly used it in their deliberations, it 
would be a powerful incentive for 
departments to collect and report the 
information. 


Many jurisdictions report on 
performance 


19.16 A growing number of 
jurisdictions in Canada have 
performance reporting initiatives. As 
we reported in 1997, several jurisdictions 
in Canada (and elsewhere) are making 
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tangible efforts to provide their 
legislatures and citizens with better 
information on how government programs 
have performed. Alberta has perhaps the 
most experience in Canada, reporting to 
the legislature on performance both 
government-wide and by ministry. 
Initiatives in Nova Scotia and New 
Brunswick that we noted in 1997 have 
continued, with some changes introduced 
by new governments. More recently, 
Quebec and British Columbia have passed 
legislation requiring the government to 
report to the legislature on performance. 
Saskatchewan also plans to enhance 
reporting. 


19.17 We did not examine the success 
of initiatives in other jurisdictions or 
compare their performance information 
with that reported by Canada’s federal 
government. We cite the experience and 
ongoing efforts of these jurisdictions to 
show that although it is not easy to 
provide better information on how 
government programs have performed, 
it can be done. 


The Expenditure Management System 


19.18 Past efforts by the federal 
government to improve reporting to 
Parliament included the creation of the 
Part II] Main Estimates in 1981. The Part 
III documents were designed to provide 
information to members of Parliament on 
departmental spending plans and on the 
performance and results produced with the 
spending previously authorized. In 1983, 
the government agreed to include 
summaries of program evaluations in the 
Part IIs. 


19.19 In 1995, the government revised 
the Expenditure Management System it 
had used to manage its affairs and report 
on them to Parliament. As part of this 
initiative, it launched the Improved 
Reporting to Parliament Project. 


19.20 From Parliament’s perspective. 
The intent of the project was to provide 
Parliament with better information on the 
government’s plans and performance. On 
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The timing of the 
spring plans and fall 
performance reports 
is intended to 
enhance the role of 


standing committees. 


a pilot basis, and with the agreement of 
Parliament, Part III of the Estimates was 
split into two documents: 


- a Report on Plans and Priorities, 
tabled in the spring with Parts I and II of 
the Main Estimates, to establish 
performance targets and outline the 
general direction the minister wants the 
department to take during the upcoming 
year and the next two fiscal years; and 


« a Performance Report, tabled in the 
fall, to indicate the extent to which results 
achieved to date reflect those that were 
planned. 


19.21 Both of these new reports are 
Estimates documents, tabled in the House 
of Commons and subject to consideration 
by the appropriate standing committees. 
The timing of the spring plans and fall 
performance reports is intended to 
enhance the role of standing committees. 
Performance reports in the fall enable 
committees, in a timely way, to report 
their conclusions to the House and to 
make their views known to the 
government as it carries out its fall 
consultations on the upcoming budget. 
Standing committees may thereby 
influence spending plans and priorities for 
the next year or beyond. The plans 
reported in the spring can provide a focus 
for committee reporting to the House on 
future years’ plans and priorities. 


Clarifying results and performance 
concepts 


19.22 Discussions of performance 
reporting can be confusing, as they 
involve a number of specific concepts and 
terms. Exhibit 19.1 presents some of 
them. 


Focus of the audit 


19.23. Our government-wide audit 
looked at the present state of performance 
reporting. This audit was in part a 
follow-up on the observations and 


recommendations we made in our 1997 
Report, Chapter 5. Our objectives were to 
assess: 


* the government’s progress in 
reporting performance information to 
Parliament, including the leadership 
provided by the Treasury Board 
Secretariat and the use of performance 
information by standing committees; and 


* the strengths and weaknesses of the 
federal regime of performance reporting. 


19.24 We examined the Part III 
Estimates documents — the reports on 
plans and priorities and departmental 
performance reports of 47 departments 
and agencies (shown in Appendix A). 
These were the larger departments and 
agencies, including the 16 that had piloted 
performance reports in 1996, and a sample 
of smaller agencies. Over 80 departments 
and agencies provide these two Estimates 
documents to Parliament annually. 


19.25 As this was a government-wide 
audit of the progress made to date in 
performance reporting, we did not audit 
the reliability or accuracy of performance 
information in reports on plans and 
priorities and departmental performance 
reports. 


19.26 Further details on our audit 
objectives, criteria and approach can be 
found at the end of the chapter in About 
the Audit. 


Observations and 
Recommendations 


The Expenditure Management 
System Since 1997 


19.27. The Improved Reporting to 
Parliament Project started in 1995 with 
five pilot departments, and expanded to 
include 16. Significant events have 
occurred since our 1997 audit: 
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+ All departments and agencies (82 in the Business of Supply endorsed the 


1999) submit reports on plans and Improved Reporting to Parliament Project 
priorities and performance reports. in December 1998, and recommended a 

+ Based on the work of the Standing number of ways for standing committees 
Committee on Procedure and House to make good use of the Estimates reports. 


Affairs, a parliamentary subcommittee on In its May 1999 response, the government 


Exhibit 19.1 


Terms and Concepts of Results and Performance 


—= 


The concept of performance deals with how well things Results Chain 


are done: 
* Are the expected results accomplished? Activities Examples 
* Are they accomplished within budget and in the most (how the program carries | negotiating, consulting, 


efficient manner? out its work) drafting legislation 


¢ Are there undue, unintended consequences? 


cheques delivered, advice 
given, information provided, 
reports produced 


Outputs 
(goods and services 
produced by the 
program) 


It also deals with whether the performance will continue 

or improve: 

¢ Is the organization learning from past experience and 
adapting? 


Intermediate satisfied users, jobs found, 


Performance, then, covers a number of ideas. 


outcomes equitable treatment, illegal 
Determining the specific aspect of performance to (the chain of entries stopped 
consequences 


Government programs undertake a number of activities 
that produce a variety of results. Programs deliver two 
kinds of results: outputs, the direct products and 


measure, and when, is not always straightforward. 
Results 
from the outputs) 


services produced by government activities; and X 

outcomes, the consequences of those outputs on | 

Canadians and our society. Outputs are results that environment improved, stronger 
managers can largely control, while the outcomes are End outcomes economy, safer streets, energy 
influenced by factors outside the programs of managers. \ (end results sought) saved 


End outcomes (sometimes called long-term, final or 
ultimate outcomes) are the end results sought. In 


between the outputs and the end outcomes, a sequence The following illustrates how the various results of an anti-smoking 
of intermediate outcomes are expected to lead to a program could be characterized. 
desired result but are not an end in themselves. Recults An Anti-Smoking Program | 
Intermediate outcomes are more easily linked to the = = : = 
activities of a program than are end outcomes. The Outputs ° anti-smoking advertisements and | 
results chain is the sequence of outputs and outcomes : 

promotions 


that occurs as a result of the activities of the program. Bani ial Gistabuisd 


¢ enforcement of smoking regulations 


Intermediate * compliance with regulations 
outcomes ¢ reduced number of smokers 
¢ reduced number of new smokers 


End outcomes ¢ reduced incidence of smoke-related 
health problems and deaths 


¢ reduced costs of health care 
associated with smoking-related 
problems 
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An important 
element of the 
revised Expenditure 
Management System 
is the move 

by departments 

to a single, 
purpose-focussed 
accountability 
structure for reporting 
financial and 
performance 
information. 


agreed to engage parliamentarians further 
by initiating Phase 2 of Improved 
Reporting to Parliament. 


« The Treasury Board Secretariat has 
established an Estimates Web site that 
includes departmental performance reports 
and reports on plans and priorities, with 
links to other performance information 
and information on cross-government 
initiatives. 


¢ In March 2000, the government 
issued Results for Canadians, a 
results-based management framework for 
the Government of Canada. 


« Asecond subcommittee established 
in May 2000 considered proposals for 
Phase 2; its report was tabled on 12 June 
2000. 


19.28 Business plans no longer 
submitted to Treasury Board. Perhaps 
the biggest change in the Expenditure 
Management System has been the 
evolution of departmental business plans. 
An integral element of the original 
Expenditure Management System was the 
requirement that departmental business 
plans be submitted to the Treasury Board 
for review, in place of multi-year 
operational plans. Business plans initially 
were to be concise, strategic three-year 
plans that set out goals, targets and 
performance measures. Several provinces, 
including Alberta, Ontario, and New 
Brunswick, have adopted business 
planning. 


19.29 Business plans have evolved 
considerably since 1997, but as of 2000, 
their submission to Treasury Board is no 
longer required. The submission process 
began to be seen as cumbersome and not 
useful. For external communication 
purposes, a number of departments use 
their report on plans and priorities as their 
business plan. 


19.30 Departments are encouraged to 
develop business plans for their own 
planning purposes. Treasury Board 


Secretariat analysts can get copies in the 
course of their ongoing dialogue with 
departments. On a selective and cyclical 
basis, any deputy minister may be asked 
to send a letter to the Secretary of the 
Treasury Board outlining current key 
issues and concerns in the department. 
The Treasury Board Secretariat intends to 
develop a yearly report to the Treasury 
Board on the key issues and risks faced by 
departments, together with options for 
dealing with them. The intent is that the 
President of the Treasury Board will 
present the results of this exercise to 
Cabinet at an early stage in the 
priority-setting cycle. 


19.31 Planning, Reporting and 
Accountability Structure still approved 
by Treasury Board. An important 
element of the revised Expenditure 
Management System is the move by 
departments to a single, purpose-focussed 
accountability structure for reporting 
financia! and non-financial information. 
This Planning, Reporting and 
Accountability Structure (PRAS) links the 
department’s internal management and 
accountability regime to its objectives, 
business lines, resource requirements and 
performance targets. It is intended as the 
basis for reporting in business plans and 
Estimates documents. Departments are 
still required to get Treasury Board 
approval for their PRAS if they want to 
significantly change their reporting 
structure. 


19.32. To provide a focus on results in 
the departmental performance reports and 
the reports on plans and priorities, 
departments are asked to prepare a chart 
of “Key Results Commitments” as a way 
of organizing their reports, and they are 
asked to identify the kind of information 
they will provide to demonstrate results. 
Work is ongoing in departments to align 
the PRAS with key results commitments 
and their business lines to ensure a link 
between expenditures and results. 
Changes in these charts are reviewed by 
Treasury Board Secretariat staff before the 
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departmental performance reports are 
completed each year. 


What We Expected in Reports 


19.33 After discussion with Treasury 
Board Secretariat officials in 1997, we 
identified five characteristics (Exhibit 
19.2) needed in performance reports to 
give a credible account of performance: 


* clear presentation of context and 
strategies; 


* clearly and concretely stated 
performance expectations; 


* performance accomplishments 
reported against expectations; 


* demonstrated capacity to learn and 
adapt; and 


¢ fair and reliable information. 


We would expect to see the first two 
characteristics in a good report on plans 
and priorities as well. 


19.34 In examining performance 
reporting by departments, we considered 
each of these elements of good reporting. 
Our findings follow, starting with our 
observations on how departments stated 
the results they expected. Next, we 
describe how they reported their actual 
results, and then we present our 
observations on the remaining elements of 
good reporting. Finally, we look at the 
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overall state of performance reporting 
after five years. 


19.35 We recognize that it may take 
time to establish expectations and report 
accomplishments in a way that is 
meaningful and useful to the organization, 
but we would expect to see significant 
progress after four or five years, and many 
examples of good performance reports. 
Where reporting is still weak, we would 
expect to see departments informing 
readers of their plans to improve it. 


Some Improvement in Stating 
Performance Expectations 


A variety of terms used to state what is 
to be accomplished 


19.36 Onsome of the key terms used in 
performance measurement and reporting, 
there is general agreement (these terms 
are set out in Exhibit 19.2). There is less 
agreement on the terms that describe other 
key elements. One important element of 
performance reporting is setting out the 
results a program is intended or expected 
to accomplish. A second key element is 
measuring the results it has actually 
accomplished. 


19.37 Stating performance 
expectations. The term “objective” is 
often used as the statement of expected 
accomplishments but it tends to hide an 
important distinction between two 
different concepts. The first concept is a 


After reading a performance report, a member of Parliament or the public should have a good 
overview of a department and how well it is doing. In particular, a reader should know the 


following: 


What are the context and the strategy? What the department does and why, what its programs 
are, what they should accomplish, what approaches will be used. 
What results were expected? What each program intends to achieve specifically over the next 


few years toward the end outcome. 


What results occurred? What has been accomplished to date, how it compared with the expected 
results, what resources were used, how the program contributed to the reported accomplishments, 
and what factors and other programs influenced the performance. 

What changes are being made? What steps are being taken to improve performance that fell 


short of expectations. 


How reliable are the data? To what extent the information reported is sound, and which data or 


information must be used with caution. 


One important element 
of performance 
reporting is setting 
out the results a 
program is intended 
or expected to 
accomplish. 


A second key element 
is measuring the 
results it has actually 
accomplished. 


Exhibit 19.2 


What Should Be ina 
Performance Report 


Report of the Auditor General of Canada — December 2000 


19-1] 


ee 
Reporting Performance to Parliament: Progress Too Slow 


Performance 
measures are the 
terms usually used to 
identify a specific 
aspect of performance 
that can be measured, 
such as the number of 
full-time jobs created. 


Progress in stating 
clear and concrete 
performance 
expectations has 
been slow overall. 


general statement of strategic direction, a 
more global statement of intent. The 
Treasury Board Secretariat has begun 
calling these the key results commitments 
— the benefits that departments have 
committed to provide to Canadians — and 
each department now sets them out in its 
Chart of Key Results Commitments. 


19.38 The second concept is a 
statement identifying specifically what is 
to be accomplished over the next few 
years. This is what our 1997 chapter 
called performance expectations. Over the 
past several years, the Treasury Board 
Secretariat and departments have used a 
number of different terms — desired 
outcomes, specific goals, intended results, 
and desired results. More recently, the 
Secretariat introduced the term planned 
results for these more specific intended 
results, as the companion term to the more 
global results commitments. We will use 
“planned results” interchangeably with 
“performance expectations”. 


19.39 Measuring performance. In 
order to report performance, some way to 
measure it is needed. Performance 
measures or indicators are the terms 
usually used to identify a specific aspect 
of performance that can be measured, 
such as the number of full-time jobs 
created or number of cheques delivered on 
time to eligible recipients. Departments 
are asked to state in the chart of key 
results commitments the performance 
measures or indicators they will use to 
demonstrate progress toward their results 
commitments. 


Setting concrete expectations remains a 
challenge 


19.40 We examined the reports on plans 
and priorities of the 47 departments and 
agencies over the last six years. We 
looked at statements of expected 
performance — both the global results 
commitments and the more specific 
planned results. Exhibit 19.3 outlines the 
assessment we made and shows the 


average annual index score across 
departments. The scores indicate that 
progress in stating performance 
expectations has been slow overall, 
levelling off in the last two years. 


19.41 However, statements of expected 
performance have become more 
outcome-focussed over the years. The 
reports for 2000-01 contained more clear 
and concrete statements of planned results 
than did those for 1996-97. Where 
progress was made, it was in more clearly 
stating expectations in terms of outputs or 
immediate outcomes and indicating a 
direction of change. There was little 
progress in specifying the timeframe for 
change and even less in stating the 
magnitude of change. Departments that 
scored highest usually stated expectations 
with both immediate and longer-term 
outcomes, and indicated the magnitude of 
change sought within a given timeframe. 
Exhibit 19.4 provides examples of clear 
and concrete performance expectations. 


19.42 Departments have progressed to 
varying degrees in setting out expected 
performance. Some departments 
improved; others fell back — their later 
reports did not set out expectations in 
terms as clear and concrete as in their 
earlier reports. For example, in its 
1997-98 Report on Plans and Priorities, 
Status of Women Canada set out, in 
reasonably clear and concrete terms, its 
performance expectations for contributing 
to equitable public policy: 


Use of gender-based analysis at the 
planning and subsequent stages in 
government policy and program 
analysis and development, by the year 
2002. 


The corresponding expectation in the 
2000-01 report, however, is less clear: 


To have directed/led/driven the 
integration of gender-based analysis 
in the policy and program 
development of selected federal 
departments and key institutions. 
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19.43 Our Office continues to call for 
clear and concrete statements of the 
results that departments plan to achieve. 
Our 1997 chapter gave examples of 
planned results that were clearly stated 
and some that were not, some that were 
concrete and others that were not. (See 
Exhibit 19.4 for more recent examples.) 


19.44 In our view, planned results need 
not necessarily be expressed as numerical 
targets to be concrete. As discussed in 
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include numerical ranges, a yes/no 
indication of a result, or a result indicated 
verbally on a comparative scale. And in 
setting out the specific result that is 
planned, it is useful to keep in mind the 
whole chain of outputs and outcomes to 
provide its context. 


Inconsistent use of concepts 


19.45 The performance reports we 
examined show widespread inconsistency 
in the content of key results commitments. 


Appendix B, concrete expectations can They range from general statements of 


To identify overall trends, we assessed each business line in the reports on plans and priorities to Exhibit 19.3 


see whether its statements of planned results were: 
Assessing Statements of 


¢ clear and concrete; Expected Performance 


¢ focussed on outcomes; and 
* plausibly linked to the business line’s activities and outputs. 


Statements of expected performance are more concrete when they can be readily measured. For 
example, statements must clearly indicate a direction of change, such as increasing the number of 
jobs created; contain specific time frames, such as by 2001; and anticipate the magnitude of 
change, such as creating 1,000 jobs. 


We combined these elements into an index to measure how well the report stated results 
commitments and planned results for each business line. An index of 10 reflected statements that 
were clear on what was expected, that specified the magnitude of change expected and the time 
frame, that were outcome-oriented, and were plausibly linked to the business line. An index close 
to zero reflected vaguely stated expectations with no apparent link to the business line. To 
calculate an index for a department, we added the business line indices after weighing them 
according to their expenditures. A given business line can rate high in some aspects of stating 
expectations and low in others. 


The chart shows the average annual score on this index across departments. 


Average Index Scores for Stating Performance 
Expectations in Reports on Plans and Priorities 


Index 
i 


1996 1997 1998 1999 2000 2001 
(pilot year) 
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Exhibit 19.4 


Statements of Performance 
Expectations - Examples 


intent (in the vast majority) to quite clear 
and concrete statements of what is to be 
accomplished, that is, planned results. In 
the results commitment chart, the content 
under “to be demonstrated” is even more 
varied and hence much less useful to 
readers. Some departments give a list of 
activities to be carried out, and others a 
list of sub-objectives; some provide a list 
of performance measures, and still others, 
a list of performance expectations. And 
we have already noted the several 


different terms used to describe expected 
performance. 


19.46 In our view, this miscellany of 
terms and concepts hinders reporting by 
confusing both the readers and the 
preparers of the reports. More uniformity 
in the types of performance expectations 
to be reported would help. The chart of 
key results commitments most recently 
requested by the Secretariat calls for two 
of the needed elements: results 


Clear and Concrete 


Maximum economic and social benefit for the global movement of people. To be demonstrated by 
“achievement of target immigration levels of 200,000 to 225,000 newcomers to Canada for 2000.” 
Citizenship and Immigration Canada, 2000-01 Report on Plans and Priorities. 


Export capability and preparedness. “ExportSource content will be improved by implementing the 
re-engineered guide, Exporter Preparation, and the new Canadian Exporter’s Guide to Intellectual 
Property, and by targeting an increase in client satisfaction from 68 percent to 70 percent.” Industry 


Canada, 1999-2000 Report on Plans and Priorities. 


The environment and human health threats posed by toxic substances and other substances of 
concern are understood, and prevented or reduced. To be demonstrated by “releases of toxins 
reported through the ARET program reduced by 25,000 tonnes by the end of 2000. Continue to meet 
mercury emissions reductions of 50% from 1990 levels in accordance with UNECE commitments.” 
Environment Canada, 1999-2000 Report on Plans and Priorities. 


To manage the Department efficiently and effectively. To be demonstrated by a “reduction in the 
Departmental vehicle fleet size by 40 percent from 1995 figures and ensure, where technically and 
operationally possible, that all new vehicles run on alternative transportation fuels, by 1998.” 
Natural Resources Canada, 1999 Departmental Performance Report. 


Unclear 


Conservation and biological sustainability of fisheries resources, marine and freshwater habitats and 
a protected environment. To be demonstrated by an “increase in public knowledge of oceans and 
freshwater issues to encourage local involvement, self-regulation, voluntary compliance and shared 
accountability.” Fisheries and Oceans, 2000-01 Report on Plans and Priorities. 


The delivery of benefits and services that contribute to the independence, quality of life and standard 
of living of Canada’s veterans. To be demonstrated by “client independence and healthy lifestyle.” 
Veterans Affairs Canada, 2000-01 Report on Plans and Priorities. 


Not Concrete 


To provide Canadians with health surveillance that anticipates, prevents and responds to health risks 
posed by diseases, food, water, drugs, pesticides, medical devices, environmental and occupational 
hazards, consumer goods, and other socio-economic determinants of health. To be demonstrated by 
“a public well informed about specific risks and benefits to their health.” Health Canada, 


1999-2000 Report on Plans and Priorities. 


Promoting the integration of newcomers. “To be demonstrated by reduced potential for fraud and 
misuse of documents through the development of profiles that can help identify potential risks to 
program integrity in the citizenship application process and enhancements to the process of issuing 
citizenship certificates.” Citizenship and Immigration, 2000-01 Report on Plans and Priorities. 
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commitments and planned results. In the 
previous chart, the column “to be 
demonstrated by” was intended to capture 
the specific indicators by which a 
department would measure and report on 
accomplishments. This element of 
reporting is still needed. After several 
years of experimenting with terms and 
presentation approaches, departments 
should now be encouraged to use this 
three-part conceptual framework — key 
results commitments, planned 
results/performance expectations, and the 
specific measures that performance is “to 
be demonstrated by”. Consistent use of 
this framework by departments would 
encourage the development of clearer and 
more concrete performance expectations 
and enable readers of performance reports 
to know what results they can expect. 


19.47. The Treasury Board Secretariat 
and departments should agree on key 
elements of performance to be reported 
and the terms the reports will use. The 
Secretariat should provide stronger 
leadership for consistent common 
practice. 
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Slow Progress in Reporting 
Accomplishments 


Improvements needed in reporting 
accomplishments 


19.48 The essence of performance 
reporting is to report what has occurred 
and relate it to what was expected. The 
fall performance reports are to set out the 
results accomplished in the period up to 
31 March of that year. Exhibit 19.5 
discusses how we assessed the reporting of 
accomplishments, and shows how well 
departments have done overall. We found 
that in general, the reporting of 
accomplishments against expectations has 
made slow progress. About a third of the 
departments did show improvement year 
by year. And while half of the departments 
did better at reporting their 
accomplishments in 1999 than in their 
first year of reporting, a quarter of them 
did worse. 


19.49 There is still a focus on outputs 
and activities. While most departments 
have been reporting more outputs over the 
years and listing fewer activities, in 1999 
only six departments focussed more on 
outcomes than on activities and outputs. 


We assessed each business line in the performance reports to 


see whether accomplishments reported were: 
* outcome-oriented; 


* compared with past performance or with that of other 


entities; 


* attributed to the program; and 


¢ plausibly linked to the business line activities and outputs. 


We combined these factors into an index to measure how well 
accomplishments were reported. Business line scores for a 
department were added after weighing them according to 
their expenditures. A given business line might rate well in 6— 
some aspects of reporting accomplishments and low in ae 
others. Lower scores tended to indicate that accomplishments 
are focussed more on activity and output, and no comparison 


or attribution is provided. 


departments produced performance reports. 


The chart shows the average index scores for reporting 
accomplishments in each of the three years when all 47 


Average Index Score for 
Reporting Accomplishments 
in Performance Reports 


Index 
i 


1999 


1997 


1998 


Movement overall has 
been toward a focus 
on outputs more than 
on outcomes, as 
departments have 
emphasized what they 
can control more than 
what they can 
influence. 


Exhibit 19.5 


Assessing How Well 
Accomplishments 
Are Reported 
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In many of the 
reports we examined, 
it was not easy to 
compare clearly 
what departments 
had accomplished 
with what they had 
planned. 


Exhibit 19.6 


Types of Accomplishments 
Reported 


Some departments had drawn back from 
reporting outcomes: 14 did not report 
outcome accomplishments in 1997, and 
16 in 1999. Movement overall has been 
toward a focus more on outputs than on 
outcomes, as departments have 
emphasized what they can control more 
than what they can influence. Reporting 
on outputs is necessary, but in the context 
of the outcomes they are meant to 
influence. 


19.50 Exhibit 19.6 shows the move 
toward reporting more outputs. However, 
the number of outputs, activities and 
outcomes reported can be somewhat 
misleading if no account is taken of the 
quality of reporting. Although the exhibit 
indicates a declining trend since 1997 in 
reporting outcomes, an analysis of the way 
some of them are reported suggests that 
this is not necessarily negative. Many of 
the “outcomes” reported earlier were 
stated in very vague terms and without 
attribution; and departments reported their 
“results” at a high level — little more than 
a restatement of their objectives, which 
they claimed to have achieved. The trend 
toward reporting more outputs could be a 
much more realistic indication of what 
departments are actually measuring. From 
reporting claimed outcomes as 
accomplishments, some departments have 
progressed to reporting measured outputs 
as accomplishments. But good reporting 
of outcomes is still a challenge for many 
departments. A number of them continue 
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to report their activities in the past year as 
accomplishments (see Exhibit 19.7). 


19.51 Too much focus on the latest 
year. Most outcomes that governments 
seek take a number of years to achieve. 
A “performance story” focussed on 
outcomes requires a discussion of the 
chain of results over several years. In 
recognition of this fact, departmental 
performance reports clearly indicate on 
the front that they cover the period up to 
31 March of that year rather than only the 
12 months in the fiscal year. 


19.52 Funds are provided annually, and 
some reporting of the outputs produced in 
the past fiscal year is quite appropriate — 
indeed, essential — to provide the context 
for the outcomes reported. However, we 
found that performance reports too often 
focus primarily or exclusively on the latest 
fiscal year. Reporting on one year 
encourages a focus on activities and 
outputs, at the expense of outcomes that 
may have resulted recently from activities 
and outputs of past years. Departments 
need to adopt a multi-year perspective on 
reporting to capture the outcomes they 
have accomplished and the related 
expenditures they have incurred. 


19.53 Departmental performance 
reports are meant to report back on the 
key results commitments and the related 
specific performance expectations stated 
in earlier reports on plans and priorities. 
Evidence of accomplishments related to 
these performance plans must be provided 
and any changes in the plans explained. In 
many of the reports we examined, it was 
not easy to compare clearly what 
departments had accomplished with what 
they had planned. There are several 
reasons: the reporting structure used often 
changes, different wording may be used, 
and presentation can be confusing. Both 
the reports on plans and priorities and the 
performance reports have undergone 
experimentation and change over the past 
few years. There is now a need for greater 
consistency between the two. 
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19.54. The Treasury Board Secretariat 
and departments should ensure that 
departmental performance reports 
consistently and clearly report back on 
the expected performance set out in the 
reports on plans and priorities relevant 
for the period ending 31 March. 


19.55 Only limited comparisons 
provided. We saw that as departments 
gained experience over the review period 
in reporting performance, there was a very 
slight increase in the number of them that 
made comparisons with previous years’ 
performance. In 1999, about half provided 
at least one comparative reference, 
although very few departments provided 
comparisons regularly and systematically. 
Almost no departments compared their 
performance with that of similar entities 
(benchmarking). Readers of a 
performance report can judge whether 
performance was reasonable only by 
comparing it with something: with the 
previously stated expectations, with past 
years’ performance, and/or with similar 
programs elsewhere. 


More effort is needed to assess a 
program’s contribution 


19.56 Program managers do not control 
the outcomes they are seeking to 
accomplish. They undertake activities that 
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produce the goods and services (outputs) 
that are expected to influence events 
toward those outcomes. As highlighted by 
Treasury Board Secretariat guidelines, it is 
therefore important that departments 
explore the extent to which their programs 
are contributing to outcomes. 


19.57 Limited discussion of the 
influence that programs have on 
outcomes. Discussion in performance 
reports of the relationship among 
activities, outputs and outcomes had 
improved little over the period we 
reviewed. Few departments discussed how 
their activities and outputs had contributed 
to the accomplishments they were 
seeking. For example, only a few included 
a logic chart or results chain that linked 
inputs, outputs and outcomes. As a result, 
readers of a report would not have a clear 
picture of the department’s contribution to 
longer-term outcomes, such as a healthy 
industrial sector or higher standard of 
living for citizens, which are often the 
kinds of end outcomes the government is 
trying to achieve. 


Few departments 
discussed how their 
activities and outputs 
had contributed to the 
accomplishments they 
were seeking. 


19.58 A few departments, such as the 
Royal Canadian Mounted Police and 
Citizenship and Immigration Canada, did 
make some attempt to discuss the 
contribution they were making, primarily 
by listing the other jurisdictions and 


Fisheries and Oceans, in its Performance Report for 1999, lists as expected performance to “provide 
Canadians with safe, efficient and accessible waterways and harbours.” It elaborates on this in a 
chart of Key Results Commitments with a long list of ways this is demonstrated — for example, 
“safe and efficient movement of marine traffic through ice-covered waters.” Icebreaking activities 
the Department reports as accomplishments include implementing a fee for icebreaking services, 
transferring the Arctic Sealift (resupplying Arctic areas) to the Government of Nunavut, working 
with the United States to expand the cost-sharing arrangements for ice patrol, and continuing an 
economic study of icebreaking benefits. The expected accomplishment, the safe and efficient 


| movement of marine traffic, is not discussed. 


Health Canada, in its performance report, lists “reduced illness, injury and death from identified 
health risks” as a desired outcome of its Management of Risks to Health Commitment. “Completed 
a revised draft Risk Management Framework” is reported as an accomplishment. This is elaborated 
later in the report through more activities stated as accomplishments in improving the risk 
management framework, such as “started the implementation of the Nutrition for Health action 
agenda” and “developed policies and programs for the safety of raw food of animal origin”. The 
Department does not discuss how these activities would contribute to or eventually lead to reduced 
illness, injury and death — the stated desired outcome. 


Exhibit 19.7 


Reporting Activities as 
Accomplishments — Examples 
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Evaluations can 
provide for better 
measurement of 
outcomes 
accomplished and 
better analysis of the 
real contribution a 
program is making. 


Exhibit 19.8 


Using Evaluation Findings to 
Strengthen a Performance 
Story - Example 


agencies that were working toward the 
same outcome. Citizenship and 
Immigration Canada describes the areas of 
co-operation with its partners, but neither 
it nor the RCMP attempts to discuss how 
each is contributing to the desired 
outcomes. 


19.59 We think it is possible to take a 
more analytical approach to describing the 
contribution made by departments. A 
results chain (see Exhibit 19.1) that 
describes the outputs produced and the 
series of intended outcomes can help to 
“tell the performance story” and the roles 
of the different players in it. By 
identifying how outputs from various 
contributors have together led to a series 
of outcomes, the contribution of each can 
be shown. Similarly, the effects of the 
program’s outputs can be presented with 
supporting evidence to demonstrate the 
case that the program played an influential 
role. 


Evaluation findings can play an 
important part in reporting 
performance but are underused 


19.60 Asmall number of departments 
included relevant results of evaluations in 
their performance reports. This was often 
useful, added information about the nature 
of the performance, its contribution to 
outcomes accomplished, and the factors 
associated with success or with 


underachievement. For example, in 
discussing a program, the RCMP’s 
performance report summarizes in “Key 
Review” boxes the results of evaluations 
or reviews related to the program. In its 
Departmental Performance Report for 
1999, Canada Economic Development for 
Quebec Regions made extensive reference 
to evaluations to identify what had 
succeeded and what had presented 
challenges. Exhibit 19.8 illustrates this. 


19.61 As Treasury Board Secretariat 
guidelines on performance reporting make 
clear, evaluations can provide for better 
measurement of outcomes accomplished 
and better analysis of the real contribution 
a program is making. A more extensive 
and systematic use of program evaluations 
would enhance performance reporting, 
increase a department’s ability to identify 
its contribution, and provide some insight 
into factors that have influenced 
performance. We noted that U.S. federal 
legislation requires agencies to include 
evaluation plans in their performance 
plans, and the findings in their 
performance reports. Including the plans 
for relevant evaluations in reports on plans 
and priorities as part of the “to be 
demonstrated by” information would 
identify more clearly the role evaluation 
will play in measuring a department’s 
results. 


A mid-program evaluation led to the following conclusions for local empowerment: : 


“The SFCQ [Special Fund for Coastal Quebec] effectively contributes to the adaptation 
and the long term economic development of fishing communities most affected by the 
crisis. It has been noted that the program provides, in a majority of cases, the _ 
reinforcement, the diversification and the stabilization of the economic fabric of 


communities. 


However, the mid-program evaluation also revealed that the development of 
entrepreneurial skills within the communities is an objective that is much more difficult to 
reach. Notwithstanding the communities’ needs in terms of consensus-building and 
animation activities, findings reveal a low level of client participation. Economic 
promotion or entrepreneurship assistance activities have not yet generated measurable 


economic results.” 


Source: Excerpt from Canada Economic Development for Quebec Regions, 1999 Performance Report. 
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19.62 Departments should include 
plans for relevant evaluations in their 
reports on plans and priorities, and any 
relevant findings from evaluations in 
their performance reports. 


Need to Improve Balance and 
Indicate Reliability of Data 


19.63 We assessed the presence of other 
characteristics of good performance 
reporting, namely, presenting a clear 
context and strategy, demonstrating a 
capacity to learn and adapt, providing fair 
and balanced information, and discussing 
the reliability of the data used. 


More balanced reporting needed to 
enhance credibility 


19.64 Departments have made little 
progress in identifying weak performance 
and showing how they have used the 
information to improve programs: the vast 
majority of performance reports contain 
no such discussion. There are exceptions, 
including Citizenship and Immigration 
Canada and the RCMP — who compare 
their accomplishments (or lack thereof) 
with their previously stated expectations, 
or discuss them transparently. However, 
most departments report only the “good 
news’, and make little mention of 
performance that did not meet 
expectations. 


19.65 Departments understandably are 
reluctant to report to Parliament and the 
public on performance that falls below 
expectations. However, selectively 
reporting only good news weakens the 
reliability of the entire performance report 
— including the good news. Members of 
Parliament have repeatedly said that they 
do not trust the credibility of reports that 
present only good news. 


19.66 What is more important, 
departmental performance reports are 
reports by ministers to Parliament on what 
their departments have accomplished with 
the funds and authorities Parliament has 
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provided to them. Ministers are 
accountable for all aspects of performance 
under their responsibility and for 
accounting fully for their stewardship of 
that responsibility. 


19.67 Report what was learned. We 
have suggested that if departments present 
performance that did not meet 
expectations in a context of what they 
now know works and does not work, they 
can significantly reduce their potential 
embarrassment. They can also indicate 
how they plan to correct the problem. This 
approach not only makes for a more 
balanced and credible performance report 
but also provides value to the organization 
in proactively identifying problem areas 
and addressing them. We noted that 
Treasury Board Secretariat guidelines do 
stress both the importance of balanced 
reporting and the need to report the 
lessons learned about what does and does 
not work. 


19.68 A good example of this can be 
found in the 1999 Departmental 
Performance Report of Citizenship and 
Immigration Canada. It notes that the 
Department fell short of its target for 
privately sponsored refugees by more than 
20 percent, for two reasons: higher rates 
of refusal and longer processing times. 
The report outlines the actions the 
Department has taken to educate groups 
and sponsors in order to reduce refusal 
rates, and the operational changes it has 
made to reduce processing times. 


More attention is needed to data 
reliability 


19.69 Good reporting requires that 
information be reliable and presented 
fairly. Readers can be misled by 
information that is not accurate enough or 
does not present a balanced picture of 
what has been accomplished. Our audit 
did not examine the accuracy of the data 
in performance reports. Rather, we looked 
to see if the reports themselves discussed 
the reliability of the data. 


Members of Parliament 
have repeatedly said 
that they do not trust 
the credibility of 
reports that present 
only good news. 


Treasury Board 
Secretariat guidelines 
stress the importance 
of balanced reporting 
and the need to report 
lessons learned. 
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The reports make little 
reference to the 
fairness and reliability 
of the performance 
information they 
contain. Very few 
reports mention the 
possibility that there 
may be shortcomings 
or problems in the 
data. 


Exhibit 19.9 


Good Practices to Improve the 
Quality of Performance 
Information 


19.70 We found little reference in the 
reports to the fairness and reliability of the 
performance information they contain. 
Very few reports mention the possibility 
that there may be shortcomings or 
problems in the data. And guidance from 
the Treasury Board Secretariat has paid 
little attention to the issue of data 
reliability. 


19.71. We noted that several other 
jurisdictions give considerably more 
attention to this issue. Alberta, for 
example, requires that departments 
include in their performance reports a 
short description of the methodology used 
to arrive at any data presented. At the U.S. 
federal level, legislation requires agencies 
to discuss the verification and validation 
of information in their performance 
reports, and legislators have publicly 
stressed the importance of data quality. 
Exhibit 19.9 provides some good practices 
followed in the United Kingdom to 
improve the quality of performance 
information. 


19.72 Our Office has pointed to a 
number of inaccuracies in the data or 
information provided in performance 
reports. For example, an audit of the 
Fisheries and Oceans fleet management 
program identified problems in the 
accuracy of information in the 
Departmental Performance Report 
(Chapter 29, Fisheries and Oceans — 
Fleet Management). In an audit of the 
RCMP earlier this year (Chapter 7), we 


found that information in the 
Departmental Performance Report was 
scant and inconsistent with information 
available in other sources. A previous 
audit of the Canadian International 
Development Agency (1998 Report, 
Chapter 21) observed that CIDA needed to 
review the information in its performance 
report more rigorously to ensure that it 
was accurate and balanced. We made a 
similar observation this year in Chapter 14 
on CIDA’s Canadian Partnership Branch. 


19.73 The Treasury Board Secretariat 
should require that departmental 
performance reports indicate the extent 
to which performance information has 
been verified or validated. 


19.74 Too early for cost information 
on results. The importance of proper cost 
information to good performance 
reporting and to managing for results 
continues to be a concern of this Office. In 
1999, Chapter 21, we reported on the 
Financial Information Strategy (FIS) 
designed to help the government 
strengthen its management of business 
lines and its accountability to Parliament. 
Departments are to integrate financial 
information with non-financial 
(operational) information and use it both 
in day-to-day management and in 
accountability reporting to Parliament. 
The 1999 audit found that most 
departments were only starting to focus on 
the FIS, despite a target implementation 
date of 1 April 2001. 


The UK National Audit Office recently issued guidance on good practices in performance reporting. 
It states that systems for collecting and validating performance data are likely to be more robust 


when agencies: 
¢ define the quality of data in advance, 


° seek advice from specialists, 


¢ establish clear performance measure definitions, 
° designate who is accountable for performance data, 
* managers are active in obtaining good quality performance data, and 


° develop and implement effective controls over the collection of data. 


Source: Good Practice in Performance Reporting in Executive Agencies and Non-departmental Public 
Bodies, Report of the Comptroller and Auditor General, UK, 2000. 
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19.75 In October 2000, we reported our 
assessment of financial management 
capabilities in selected departments. A 
common gap we found again was the 
limited ability to integrate financial and 
non-financial information. Most 
departments indicated that they had only 
begun the task of integrating financial and 
operational systems. In our latest audit of 
the federal initiative at managing for 
results (Chapter 20 of this Report), we 
also found a poor link between results and 
cost information. Our examination of 10 
program areas found that limited cost 
information was available to departmental 
managers. 


19.76 This gap has clear implications 
for individual departments and for the 
government as a whole. Until they are 
able to integrate information on costs and 
on results, departments cannot give 
Parliament and Canadians a realistic 
picture of what it has actually cost to 
produce a given result. In addition, senior 
management cannot readily forecast the 
costs of major decisions, like a decision to 
increase or decrease the level of service a 
program provides. 


19.77 Performance reports generally 
state expenditures by business line, 
indicating the cost of producing all the 
business line’s results. However, a few 
departments have included some spending 
information by sub-business line or by key 
results commitment. The RCMP, for 
example, gives inputs (finances and staff) 
for some of its business line components, 
while Natural Resources Canada gives the 
costs of selected components. The 
Treasury Board Secretariat has a project 
under way for departments to share their 
practices for linking financial and cost 
information to results. 


19.78 The Treasury Board Secretariat 
should enhance its efforts to identify 
and share with departments practical 
approaches to linking financial and 
non-financial information. 
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Performance Reporting Shows 
Modest Improvement 


19.79 As we have mentioned, initiatives 
to improve reporting to Parliament have a 
long history. The 1980s saw the 
introduction of the Part III of the 
Estimates, which moved reporting toward 
a purpose-based focus. The current 
performance reports are a significant 
improvement over those early efforts. And 
even in the past few years, departments 
have considerably improved the 
presentation of their performance 
information. Most show a good effort to 
situate the department and many are quite 
reader-friendly. 


19.80 It also is clear, as we observe in 
Chapter 20 of this Report, that the concept 
of “results”, including outcomes, has 
become widely accepted as the basis for 
reporting performance. The structure of 
reporting is clearly based on a 
department’s Chart of Key Results 
Commitments. These charts form the basis 
for setting out the expected performance 
and reporting the results actually 
accomplished. The culture seems to have 
become more focussed on results, even if 
full implementation has yet to come. 


Some departments show progress in 
communicating performance 


19.81 No department we reviewed was 
exceptional in reporting its performance; 
our review showed that the practice is still 
evolving. Some departments are still 
trying to find a workable format and 
approach, and no overall “best practice” 
or “preferred model” for performance 
reporting emerged among the departments 
we reviewed. 


19.82 While we did not examine the 
accuracy of the data they used, we did 

note that a number of departments have 
made steady progress in reporting their 
performance over the last several years: 


+ Atlantic Canada Opportunities 
Agency 


Until they are able to 
integrate information 
on costs and on 
results, departments 
cannot give Parliament 
and Canadians a 
realistic picture of 
what it has actually 
cost to produce a 
given result. 


Departments have 
considerably improved 
the presentation of 
their performance 
information. Most 
show a good effort to 
situate the department 
and many are quite 
reader-friendly. 
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« Canada Economic Development for 
Quebec Regions 


« Citizenship and Immigration Canada 
« Royal Canadian Mounted Police 


¢ Western Economic Diversification 
Canada 


Their reports offer several examples of 
good reporting practice, particularly in 
communicating performance. 


19.83. Good practices in some areas 
but weaknesses in others. In our view, 
the better performance reports stated, at a 
level below the key results commitments, 
what the departments had expected to 
achieve and then reported against these 
specific expectations what they actually 
achieved. The five departments we have 
indicated did this to different degrees, 
using different approaches. Atlantic 
Canada Opportunities Agency, for 
example, set out in a short table the 
general aim of each activity area, 
described the specific result expected, and 
stated the result achieved, followed by a 
short discussion. 


19.84 The RCMP’s Departmental 
Performance Report for 1999 provided for 
each program area a well-laid-out, 
one-page discussion of its planned results 
(such as contributing to a decline in 
violent crime), an outline of the problem, 
the strategies used, its partners, the actual 
results (outputs and outcomes) and its 
planned improvements. The performance 
story was quite clear. More concrete 
statements of expectations would improve 
the report, however. 


19.85 Citizenship and Immigration 
Canada (CIC) has adopted a “performance 
at a glance” approach. Before the section 
on accomplishments it states the 
performance that had been expected in 
each area and indicates whether that 
performance was achieved or not. This 
would be a good method for reporting 
forthrightly on performance that is less 
satisfactory than expected. We note, 


however, that the expected performance 
could be stated in-the CIC report in clearer 
and more concrete terms with a stronger 
focus on outcomes, which would allow the 
reader to judge whether the actual 
performance is what was expected. 


19.86 Reports of other departments 
had good features. A number of other 
departments had innovative reporting 
practices. For example, Natural Resources 
Canada has experimented over the years 
with different approaches to reporting. 
One section of its 1999 Departmental 
Performance Report features tables of 
selected indicators that show the 
performance expected, the trend and the 
current status of performance for each 
indicator, a discussion of its significance 
and its relationship to Natural Resources 
Canada programs, and what the 
Department is doing to improve 
performance in that area. These tables are 
clear and easy to read, but they need to be 
integrated into the overall performance 
story the Department is telling about each 
of its goals and objectives. 


19.87 Agriculture and Agri-Food 
Canada has also adopted a reader-friendly 
approach to performance reporting. Its 
1999 Performance Report features a 
colour pull-out with a “report card” listing 
its key results commitments, and it has 
checked off those that were met. While 
the format is easy to read and the report is 
concise (especially given that it is a large 
department), the information it provides is 
highly aggregated and includes no clear 
and concrete performance expectations. 
Thus, it is difficult to determine whether 
the Department is performing well or not. 


Good performance reporting overall is 
still some years away 


19.88 Based on our examination of 
their performance information, the 

47 departments together have made only 
slow progress since our 1997 audit in 
reporting their performance. Some 
departments have improved some aspects 
of reporting, but on the whole, the 
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somewhat optimistic picture we presented 
in 1997 has not materialized. 


19.89 The information the 

47 departments presented in their reports 
on plans and priorities and their 
performance reports varied widely in 
quality. Some had made little progress 
over the three or four years; others had 
improved but had then fallen back below 
the level of their earlier reporting. Large 
and small departments differed little in the 
extent to which they improved their 
reporting. 


19.90 Pilot departments did no better. 
We looked at the progress made by the 
16 pilot departments (see Appendix A) 
that first reported in 1996, and compared 
it with the progress of other departments. 
Overall, their extra year of reporting 
experience and the additional flexibility 
they were provided did not put the pilot 
departments ahead of the others in the 
quality of their reporting. As a group, the 
pilots made uneven progress, and many 
other departments that had started a year 
later progressed further. 


19.91 Some departments have not 
progressed. Several departments seem to 
have a lot of trouble “telling the 
performance story” in their performance 
reports. Many of their performance reports 
are little more than descriptions of 
activities carried out over the year. 
Performance expectations are vaguely 
stated and focus on activities, and they 
report as “accomplishments” the mere 
completion of those activities. Some of 
these reports have a section on how the 
department provides benefits to 
Canadians, but it is not linked to the 
department’s activities and outputs and 
there is no discussion of intermediate 
outcomes so a reader can clearly see the 
connection. 


19.92 Recommendations to 
departments in 1997 are still valid. The 
recommendations in our 1997 chapter on 
performance reporting are listed in 
Appendix C, which also indicates the 
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extent to which they have been 
implemented. Overall, progress in 
addressing our recommendations has been 
unsatisfactory. The recommendations are 
still valid; the observations and 
recommendations in this chapter focus on 
further important areas that need to 
improve. 


Large and small 
departments differed 
little in the extent to 
which they improved 


Treasury Board Secretariat their reporting. 


Leadership for Reporting 


19.93 Numerous efforts made over 
past years. The Treasury Board 
Secretariat has taken a relatively strong 
stance in promoting improved reporting to 
Parliament and, through an interactive and 
ongoing process, in developing 
departmental capacity for reporting. The 
Secretariat’s efforts include the following: 


* a Web site that contains a full array 
of past years’ call letters, guidelines and 
reports (both reports on plans and 
priorities and departmental performance 
reports); 


* regular information sessions (which 
were particularly well attended) when the 
Improved Reporting to Parliament Project 
was first introduced; 


* the production of periodic 
question-and-answer newsletters; 


* a symposium for practitioners in 
each of the past three years; 


The Treasury Board 
Secretariat has taken 
a relatively strong 
stance in promoting 
improved reporting 
to Parliament. 


* peer review exercises; and 


* internal technical assessments of 
reports. 


19.94 We asked a few departments 
about the leadership shown by the 
Treasury Board Secretariat. They were 
generally satisfied with the Secretariat’s 
support and leadership. The pilot 
departments particularly appreciated the 
flexibility they were given to try different 
reporting approaches. Yet there is a lack 
of practical guidance on reporting specific 
issues, like the performance of 
policy-making units and regulatory 
programs. Departmental officials said they 
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The Treasury Board 
Secretariat and some 
departments are 
working on innovative 
electronic reporting or 
e-reporting. 


would have appreciated more guidance on 
setting targets for results in such areas and 
defining appropriate indicators of 
performance. 


19.95 Nevertheless, as already noted, 
progress by departments overall during 
these initial years of experimenting has 
been slow. Later, we discuss some 
possible reasons for this slow rate of 
progress, but it may be partly due to a lack 
of discipline in the reporting regime and 
the need for a stronger role by the 
Treasury Board Secretariat. As we have 
suggested, there are considerable 
differences in the way departments 
approach the basic elements of reporting 
like stating expectations clearly and 
concretely and reporting specifically 
against them. 


Better electronic reporting needs to be 
encouraged 


19.96 The Treasury Board Secretariat 
and some departments are working on 
innovative electronic or e-reporting. The 
Secretariat’s Web site for the Improved 
Reporting to Parliament project provides a 
user-friendly guide to performance 
reporting. Users can access electronic 
versions of all departmental performance 
reports, reports on plans and priorities, 
and reports from the President of the 
Treasury Board for the current and 
previous year. They can also find an index 
of all references in performance reports to 
selected cross-department issues or 
government priorities, and an explanation 
of the Estimates process and the context in 
which these reports are produced. 


19.97 Some departments have adopted 
their own innovative approaches to 
reporting through electronic media. 
Agriculture and Agri-Food Canada and 
Citizenship and Immigration Canada, for 
example, have each produced a more 
concise report by providing Web site 
references to more detailed information. 
This means that a reader can get more 
information on a particular area of interest 
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by clicking on the appropriate Web 
address. A number of departments present 
their performance reports in a summary 
form that the reader can use to access 
details on particular issues through hot 
links. However, most departments simply 
post their performance reports on their 
Web sites. 


19.98 More attention to e-reporting is 
needed. Electronic reporting solves the 
problem of providing both high-level 
summary information and more detailed 
information. Both serve various needs of 
parliamentarians and the public. 
Electronic reporting also opens the 
possibility of more timely analysis as well 
as horizontal and longitudinal analysis. 
This is clearly the direction of the future. 


19.99 The Treasury Board Secretariat 
should encourage departments to share 
their experience in electronic reporting 
of departmental performance, and 
provide them with guidance on making 
their performance reports more concise 
and user-friendly. 


Treasury Board Secretariat is focussing 
on collective reporting 


19.100 In addition to its efforts to foster 
departmental reporting, the Treasury 
Board Secretariat has taken important 
steps to support collective reporting — 
reporting on performance in an area that 
involves more than one department or 
government. In recent years, it has taken 
the lead in government to develop a 
performance measurement framework for 
the Social Union Framework Agreement; 
has provided assistance to departments in 
developing their accountability and 
reporting frameworks; and has ensured 
that all Treasury Board submissions for 
funding of interdepartmental initiatives 
include a requirement for strong 
performance reporting. Lead departments 
of interdepartmental initiatives have 
indicated that they value the Secretariat’s 
contribution. 
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19.101 The Treasury Board President’s 
report provides valuable information. 
Since 1995, the President of the Treasury 
Board has tabled an annual overview 
document in Parliament. The first one, 
Strengthening Government Review, 
described how the government was 
managing, using and strengthening review. 
Subsequent annual reports have focussed 
more broadly on managing for results and 
reporting performance information, and 
have been tabled in Parliament along with 
all the departmental performance reports. 


19.102 The President’s reports are 
intended to provide an update on progress 
made by the government overall toward 
managing for results and reporting to 
Parliament on results. They have 
described government strategies for 
measuring and reporting on departmental, 
collective and societal results: 


* In 1999, the report enhanced the 
information in the set of departmental 
performance reports tabled in Parliament 
by adding a compendium of all 
departments’ charts of key results 
commitments. 


« Successive reports have explored the 
presentation of performance information 
under each priority mentioned in the 
Speech from the Throne. 


* Collective results are reported from 
different perspectives each year in 
response to requests from 
parliamentarians. 


+ Work to develop societal indicators 
is ongoing. The 1999 report made a 
significant contribution by presenting 
possible indicators to measure 
performance in three areas: Health, 
Environment and Public Safety; Economic 
Opportunity and Participation; and Social 
Participation and Inclusion. 


19.103 However, it is true, as noted in 
the President’s 1998 report, that “the 
government is not ready to give a 
comprehensive overview of 
government-wide performance.” And the 
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President’s report itself is evolving, 
changing its focus and presentation every 
year. It does not report consistently what 
has been accomplished toward 
commitments made in previous years. 


19.104 Nevertheless, important steps 
have been taken toward government-wide 
performance reporting. The challenge is to 
build on these initiatives, developing 
collective and societal indicators and 
presenting meaningful government-wide 
information in a convenient format for 
Parliament. 


More Involvement of 
Parliamentarians Ils Needed 


Use by parliamentarians still limited 


19.105 In 1997 we reported that the 
review and use of the new Estimates 
documents by standing committees was 
limited. This has not changed, essentially. 
Some committees have carried out 
reviews but most have not examined the 
Estimates documents. A major stumbling 
block identified frequently and noted in 
our 1997 Report is parliamentarians’ 
frustration over the limits on the role they 
can play in the Estimates process. 


Improvements proposed to the current 
regime for reporting to Parliament 


19.106 Parliamentary committees have 
examined reporting to Parliament. 
Since our Report in 1997, the Standing 
Committee on Procedure and House 
Affairs has tabled two reports to the House 
of Commons that looked at reporting to 
Parliament. In December 1998, the 
Committee made a number of 
recommendations for new structures, 
tools, and incentives to improve the 
business of supply (see Appendix D for a 
synthesis of the recommendations and the 
government’s response). It also firmly 
supported improved reporting to 
Parliament and stronger government 
accountability for Estimates matters. Two 
innovations that were suggested are 
noteworthy: the creation of a new standing 


Some parliamentary 
committees have 
Carried out reviews 
but most have 

not examined the 
Estimates documents. 


The Standing 
Committee on 
Procedure and House 
Affairs firmly 
supported improved 
reporting to Parliament 
and stronger 
government 
accountability for 
Estimates matters. 
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Members made a 
number of comments 
and suggestions for 
improving reports to 
Parliament. 


committee on Estimates; and amendment 
of Standing Orders to give standing 
committees the ability to reallocate funds 
as an incentive to study the Estimates 
documents. 


19.107 In its response, the government 
recognized the merits of the 
recommendations but did not support the 
substantial changes they implied. 
However, it agreed with the objective of 
improved reporting to Parliament and 
supported the continuation of its Improved 
Reporting to Parliament Project. 


19.108 In its Thirty-Seventh Report 
tabled in June 2000, on Improved 
Financial Reporting to Parliament, the 
Standing Committee on Procedure and 
House Affairs made further 
recommendations. A subcommittee was 
struck to consider a number of proposals 
made by the government: 


* Tailored Estimates information for 
Parliament. Performance plans would 
cover a three-year period. Performance 
reports would be based on key results and 
not processes, would disclose all relevant 
information, and would be based on 
government-wide costing standards. There 
would be increased reporting of results 
that involved more than one jurisdiction. 


* Streamlined and consolidated 
reporting for Parliament. The 
government would enhance electronic 
access to information, consolidate 
reporting where possible, and make use of 
executive summaries. 


* Strengthened parliamentary 
review. The role of standing committees 
would be strengthened by having them 
review the key results commitments of 
departments. The government would 
continue improving the Estimates 
reporting process, and encourage standing 
committees to review reports on plans and 
priorities and departmental performance 
reports. 


- Improved reporting of 
expenditures by use. This proposal would 
consider integrating accrual accounting in 
the budgetary process and improve the 
spending information provided to 
Parliament in the Main and 
Supplementary Estimates. 


19.109 For the most part, the 
subcommittee recommended that the 
government’s proposals be accepted. In 
addition, it recommended that 
consideration be given again to 
establishing a standing committee on the 
Estimates, that more resources be 
provided to standing committees for 
review of Estimates documents, and that 
the reporting of activities or programs that 
span more than one department be 
consolidated. In testimony before the 
committee, this Office agreed in general 
with the Treasury Board Secretariat’s 
proposals. Further detail on these 
proposals and the subcommittee’s 
recommendations are in Appendix E to 
this chapter. 


Members’ suggestions for improvement 


19.110 We interviewed a number of 
chairs and vice-chairs of standing 
committees as well as staff and 
researchers. We also looked at the reports 
of a number of roundtables on reporting to 
Parliament that had involved members, 
and at the results of several Treasury 
Board Secretariat surveys of members of 
Parliament. Members made a number of 
comments and suggestions for improving 
reports to Parliament and for their 
subsequent review: 


« Changes are needed in the process 
for parliamentary review of the Estimates 
if it is to become more meaningful. Some 
members pointed to the recommendations 
made by the Subcommittee on the 
Business of Supply. 


¢ The new Estimates documents are 
still not very user-friendly. Information on 
programs of interest to members can be 
hard to find, and tracking information 
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from year to year can be difficult or 
impossible. 


* Members need better dialogue with 
officials on reporting. One suggestion was 
that committees be more involved in the 
planning of performance reporting by, for 
example, reviewing performance 
expectations and results commitments. 


+ Departmental performance reports 
are not balanced, presenting only the 
“good news”. As a result, their credibility 
is compromised. 


+ Another reporting weakness is the 
vagueness of performance expectations 
that “you can’t hang your hat on.” 


Changes Are Required if 
Performance Reporting Is to 
Improve 


The reporting regime has strengths 


19.111 In reviewing the design and 
implementation of current reporting to 
Parliament, we noted a number of 
strengths: 


+ The clear intent is to focus strongly 
on reporting outcomes — the 
consequences of programs and services 
that matter to Canadians. 


+ All departments and agencies are 
reporting on their performance, and have 
done so now for three years. 


+ There is momentum: progress is 
being made, albeit slowly, and the 
government has made numerous efforts to 
engage parliamentarians on the issue of 
reporting. 


* Government clearly sees the 
reporting initiative not as a singular effort 
but as part of meeting a larger need to 
manage for results, a key objective of its 
reform efforts. 


19.112 Nevertheless, we found that 
progress has not been enough to ensure 
that departments and government will 
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arrive at good performance reporting 
anytime soon. Clearly, along with some 
strengths there are a number of 
weaknesses in the current regime. 


What is the problem? 


19.113 Performance reporting has a 
long way to go. While we understand that 
good reporting takes time to develop, it is 
reasonable to expect that in the three to 
five years that departments have had, an 
organization dedicated to improving its 
performance reporting could have made 
significant progress. We were 
disappointed by the pace of the federal 
government’s progress. 


19.114 As we have noted, several 
departments have made steady progress 
each year, though they still have some 
way to go, and we would expect to see the 
same progress in the vast majority of 
departments and agencies. The uneven 
progress of some — the weakening of 
reporting, even — is a concern. At the 
current rate of progress, it will be many 
years before good performance reporting 
becomes routine. It is taking too long. 


19.115 We had expected to see better 
progress and had not set out to try and 
explain the lack of progress. Nevertheless, 
three factors stand out as contributing to 
the current state of reporting: 


* Too frequently, basic principles of 
good reporting are not understood or 
applied. 


+ Performance reporting takes place in 
a political environment. 


+ There have been no incentives 
available for good reporting practices or 
sanctions applied for bad. 


19.116 Basic reporting principles not 
well accepted. As we have already 
observed, there is frequently confusion or 
disagreement over basic principles of 
reporting. For example we found the 
following: 


* confusion over, and inconsistent 
interpretation of, what is needed to 


We found that 
progress has not been 
enough to ensure that 
departments and the 
government will arrive 
at good performance 
reporting anytime 
soon. 


There is frequently 
confusion or 
disagreement over 
basic principles of 
reporting. 
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An important feature 
of performance 
reporting and 
managing for results is 
that those involved 
use similar concepts 
and similar terms to 
describe the key 
elements of reporting, 
such as performance 
expectations. 


Reporting to 
Parliament on the 
performance of 
government programs 
and services takes 
place in a political 
environment that is 
frequently adversarial. 


articulate results commitments, 
performance expectations and 
performance measures, 


* persistent attempts to report 
performance for only the latest fiscal year 
rather than the more appropriate period, 
“up to 31 March of ...”; and 


¢ infrequent recognition of the 
principle that performance reporting 
requires a meaningful comparison with 
previous levels of performance, with 
performance of similar programs 
elsewhere and/or with previously 
determined performance expectations 
stated clearly and concretely, so that 
readers can assess not only what was done 
— the results accomplished — but also 
how well it was done — how close it 
came to the intended outcome. 


19.117 Poor performance reporting is 
safe reporting. In our view, perhaps the 
chief drawback at present is that an 
organization that does not work to 
improve its reporting suffers no adverse 
effects as a result. It may receive 
unfavourable comment from its 
parliamentary committee and some 
pressure to improve by mid-level officials 
in the Treasury Board Secretariat, but 
essentially departments can meet 
minimum requirements and not devote 
much attention to reporting. Reporting 
only a list of various activities carried out 
is a safe route to follow, ensuring for the 
most part that officials and the minister do 
not have to answer any tough questions 
about the department’s performance as a 
result of its formal reports to Parliament. 


Clear and common reporting principles 
are needed 


19.118 In 1997, this Office outlined its 
expectations for good reporting (see 
Exhibit 19.2). We used them as criteria in 
this audit. The Treasury Board, too, has 
set out guidelines for reports on plans and 
priorities and departmental performance 
reports that are quite consistent with our 
expectations. There are also guidelines for 


departments on setting out the underlying 
framework for reporting results, namely 
the Planning, Reporting and 
Accountability Structure, which is 
approved by Treasury Board. All of these 
guidelines, however, have not been clear 
and consistent over the years and 
departments have used them 
inconsistently. As a result, it is likely that 
departments currently use a variety of 
criteria to determine what constitutes good 
performance reporting. Further, what has 
so far not been done on a formal basis is 
to engage the users of the performance 
reports — specifically, members of 
parliamentary committees — in 
stipulating what they want and expect 
performance reports to provide. Given the 
record to date, it may be time to move 
toward agreed reporting principles. 


19.119 As we pointed out in 1997, an 
important feature of performance 
reporting and managing for results is that 
those involved use similar concepts and 
similar terms to describe the key elements 
of reporting, such as performance 
expectations. 


19.120 We note that the CCAF has work 
under way with the Canadian Council of 
Legislative Auditors and senior officials 
of a number of Canadian governments, 
including the federal government, to 
explore principles of good reporting. 


19.121 The government should work 
with Parliament and its auditor to agree 
on principles for reporting performance 
information to Parliament. 


Performance is reported in a political 
environment 


19.122 Reporting to Parliament on the 
performance of government programs and 
services takes place in a political 
environment that is frequently adversarial. 
Expectations for full and balanced 
reporting need to be tempered with the 
realization that what may be seen as a 
“report card” on government is a recent 
feature of public accountability. The intent 
of performance reporting is to enhance 
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debate in Parliament and the public at 
large on the workings of government. But 
clearly, there is a risk that performance 
reporting could become a political tool of 
ministers or the opposition rather than a 
source of better information. At present, 
there is little incentive to report the “bad 
news’. Performance reporting does not yet 
play a significant role in Parliament. It 
will take time for balanced performance 
reporting to become a useful factor in 
parliamentary debate and scrutiny. 


19.123 The more that reporting is seen to 
provide a balanced and factual account of 
how well programs are working and what 
is learned when things do not work, the 
more effectively it can enhance 
parliamentary debate and build trust in 
government. The more routine that 
performance reporting becomes, the 
sooner it is likely to be accepted as a 
normal part of public accountability. The 
more that performance reports are seen to 
withhold information, the greater will be 
their vulnerability to political use. 


More consideration of incentives needed 


19.124 An award for good reporting is 
a good idea. The Treasury Board 
Secretariat is planning to present several 
awards each year for the best performance 
reports. An outside committee has been 
identified and the first awards were 
planned for fall 2000. In our view, this is a 
good step and provides some 
encouragement to departments to improve 
their reporting. It parallels the Auditor 
General’s Award for Excellence in Annual 
Reporting by Crown Corporations, now in 
its seventh year. 


19.125 Good reporting is based on 
good managing for results. The 
information that departments include in 
their performance reports should come 
from the information systems they use to 
manage. The more a department is 
managing for results, the more likely it 
will be to have good information for 
external reporting. Chapter 20 discusses a 
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number of factors that can encourage a 
focus on results and hence improve 
reporting. 


19.126 A role of assurance. While 
departments must be responsible for the 
integrity of their data, reporting regimes in 
other jurisdictions have often included the 
provision of assurance in the role of their 
external auditors. In Alberta, the Auditor 
General audits ministries’ reporting of 
methodology and, at the same time, 
reviews their information systems and 
whether they are compatible with the 
ministries’ internal control frameworks. In 
New Zealand, the Audit Office provides 
assurance on the fairness of reported 
performance achievements, while the 
audit office of Western Australia provides 
assurance that performance indicators are 
relevant and appropriate. The Texas audit 
office audits performance data for 
accuracy, while in Florida, performance 
measures proposed by state agencies are 
reviewed by a legislative office for 
validity and reliability. 


19.127 At the federal level in Canada, 
the Office of the Auditor General assesses 
the performance information in the annual 
reports of three newly created agencies 
and provides assurance of its fairness and 
reliability. 


19.128 A role for internal audit. 
Modern comptrollership contends that 
internal audit should audit and verify the 
accuracy and completeness of 
performance information. To date, 
however, this role for internal audit has 
not been substantively addressed. The 
procedures for verifying performance 
information are still not well defined, and 
internal audit units in departments 
generally lack the needed skills and 
experience. A draft Treasury Board policy 
on internal audit does call for it to play 
this assurance role. We agree with that 
policy. Internal audit ought to be the first 
place a department turns to for assurance 
of its data quality. 


The information that 
departments include in 
their performance 
reports should come 
from the information 
systems they use to 
manage. 


Internal audit ought to 
be the first place a 
department turns to 
for assurance of its 
data quality. 
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Setting and 
challenging 
performance 
expectations is an 
integral part of 
effective planning and 
priority setting. 


19.129 The Treasury Board Secretariat 
and departments should develop the 
capacity of internal audit to provide 
assurance to deputy heads on the 
quality of information reported to 
Parliament and information used for 
decision making. 


A stronger challenge of expectations is 
required 


19.130 In some jurisdictions, 
departmental performance expectations 
are challenged externally. We note that 
in many other jurisdictions, departments’ 
statements of expected performance are 
subjected to a degree of challenge from 
outside the organization. Government 
caucus members of the legislature in 
Alberta and Ontario have been involved in 
reviewing performance expectations and 
setting targets, as well as reviewing 
reported results. In other cases, such as the 
U.S. and the states of Florida and Texas, 
legislative and executive oversight bodies 
review and challenge agencies’ 
performance expectations and specific 
targets for performance as part of the 
budgetary approval process. 


19.131 Treasury Board Secretariat 
review is limited. Treasury Board 
Secretariat officials provide some level of 
challenge to departments but it is informal 
and limited. Questions typically come 
from the Secretariat’s liaison officer for 
performance reporting, not the Treasury 
Board analyst who is responsible for 
reviewing departments’ requests for 
funding and for program changes and 
assessing program integrity (identifying 
strategic program issues and the critical 
risks to the continued achievement of 
results). The Secretariat may not always 
be in a position to question a quantity 
specified as a planned result — the 
magnitude of the change expected. But as 
a management board it could usefully 
challenge the way departments have stated 
their performance expectations and how 
they arrived at them. Results reported 
against clearly stated planned results 


would thus be a key element in the 
Secretariat’s assessment of departmental 
programs. 


19.132 Parliamentary review of 
expected performance would be 
beneficial. There would be an advantage 
to Parliament’s involvement in reviewing 
departments’ statements of expected 
performance. This review could be after 
the reports on plans and priorities are 
tabled in the spring, or part of a review of 
performance reports in the fall; it would 
provide a healthy challenge to 
departments. Involvement of 
parliamentarians in articulating expected 
performance would likely motivate 
departments to report back to Parliament 
fairly and reliably. It would also provide 
an incentive for parliamentarians to 
review and use the Estimates information 
more fully. 


19.133 Equally important, however, 
would be an effective challenge within 
departments of both the performance 
expectations and the actual results 
reported back against them. This would 
normally involve the senior management 
of the department. Setting and challenging 
performance expectations is an integral 
part of effective planning and priority 
setting. If departments are to manage for 
results, they need to plan for realistic and 
challenging results and to state them as 
clear, concrete and time-specific 
performance expectations that all senior 
managers accept. In turn, the results 
reported against the expectations first 
need to undergo scrutiny by senior 
management, not only to assure the 
validity of the information but also to 
reaffirm its relevance. If the results 
achieved are not meeting the performance 
expectations, changes in the program may 
be needed, or the planned results may 
need to be reconsidered. 


19.134 The Treasury Board Secretariat 
should provide a stronger challenge to 
departments’ statements of expected 
performance. 
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19.135 The government should 
encourage departments to discuss their 
performance expectations with the 
related standing committees. 


A stronger role for parliamentary 
committees 


19.136 We suggested in 1997 that 
standing committees might wish to 
strengthen their review of Estimates 
documents. In February 1998, we 
provided all members with Parliamentary 
Committee Review of the Revised 
Estimates Documents, a report that 
described the Estimates system and 
offered suggestions for carrying out such a 
review. We still believe that a 
strengthened review would be useful for 
committee work and would influence 
departments to improve the performance 
information they provide. 


19.137 Mechanisms for reporting to 
the House are important. In 1997, we 
noted that new arrangements were needed 
to facilitate reporting by standing 
committees on the fall departmental 
performance reports. At present, the rules 
of the House provide for committee 
scrutiny of the Main Estimates as part of 
the business of Supply. The committees 
also need the reports on plans and 
priorities in order to understand the votes 
in the Main Estimates. Since 1994, the 
rules have allowed committees to consider 
and report on those plans and priorities. In 
addition, a standing committee can study 
and report on any matter relating to the 
mandate, management and operation of 
the departments and agencies it oversees. 
While there is nothing to preclude 
committees’ considering departmental 
performance reports and reporting their 
findings to the House, there is no rule 
explicitly empowering them to do so. The 
adoption of an explicit Standing Order 
would indicate the House’s willingness to 
receive and consider committee reports 
based on the examination of departmental 
performance reports. It would encourage 
committees to hold more hearings and 
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report more often to the House. For 
example, the House adopted a specific 
Standing Order in 1994 to empower the 
Standing Committee on Finance to hold 
pre-Budget consultations. 


19.138 Parliament may wish to 
consider an enhanced role for its 
committees in reviewing Estimates 
documents, including review of the 
statements by departments of their 
expected performance. 


Is it time for legislation? 


19.139 As we have noted, in 1995 the 
government revised the Expenditure 
Management System and initiated the 
Improved Reporting to Parliament project 
to improve the Estimates documents. To 
date, these reforms have not been 


established in legislation. We believe that a 


strengthened 
parliamentary review 
would be useful for 
committee work and 
would influence 
departments to 


19.140 Under the Financial 
Administration Act, the Treasury Board is 
responsible for all matters related to 
financial management, including the 
Estimates. The Treasury Board Secretariat 
has issued guidance to departments, 
requiring that financial and non-financial 
performance information be provided to 
Parliament. However, the form and 


content of the Estimates documents are improve the 

not specified in legislation, and may be 

changed by the government, although in performance 

the past the government has also sought information they 
Parliament’s agreement. provide. 


19.141 Parliament agreed in 1997 to split 
the Part III Estimates into the reports on 
plans and priorities and the departmental 
performance reports, but it did so only on 
a pilot basis. For the past three years, all 
departments have provided performance 
information to Parliament but there is no 
certainty that parliamentarians will 
continue to receive this type of 
information from departments after the 
pilot project ends. 


19.142 In June 2000 the Standing 
Committee on Procedure and House 
Affairs recommended that parliamentary 
review be strengthened and, in particular, 
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In our view, it is time 
for the government to 
consider introducing 
accountability 
legislation. 


Exhibit 19.10 


that the government reconsider the 
recommendations in the Committee’s 
1998 report on the subject. In the earlier 
report, the Committee had endorsed the 
pilot project to improve reporting to 
Parliament and had recommended that 
standing committees make full use of the 
information on departmental plans and 
performance in their study of the 
Estimates. The Committee considered that 
good performance reports should increase 
Parliament’s ability to know what the 
government is accomplishing with the 
dollars and authorities granted to it and to 
effectively challenge government plans 
for priorities and spending in the future. 


19.143 It is clear that good reporting on 
performance can increase Parliament’s 
ability to scrutinize public expenditures. 
In light of the modest progress to date, we 
think a stronger and more politically 
visible initiative is needed, one that will 
focus the attention of parliamentarians and 
officials on reporting. In our view, it is 
time for the government to consider 
introducing accountability legislation. 


19.144 Legislation alone will not ensure 
good reporting. But it could serve as a 
signal that Estimates reform is not a 
passing fad. The enactment of legislation 
would involve parliamentarians in 


Accountability Legislation in Other Jurisdictions - Examples 


Western Australia 


Jurisdiction Accountability Legislation 

Canada 

Alberta Government Accountability Act, 1995 

British Columbia Budget Transparency and Accountability Act, 2000 
Quebec Public Administration Act, 2000 

Australia 


Financial Administration and Audit Act, 1985 


designing a regime suitable to their needs. 
Moreover, legislation can secure further 
progress in a number of ways. It can: 


¢ address concerns about the staying 
power of the federal reporting initiative. 
After five years of experimenting, 
legislation would ensure continuity over 
successive governments by formalizing 
the arrangement’s permanence. 


* provide an institutional anchor for 
results-based reporting initiatives, giving 
them standing to help ensure that they will 
continue to be taken seriously. The present 
initiative has clear objectives and some 
momentum, but it needs discipline and 
drive (see paragraph 19.111). 


¢ provide Parliament with a focus for 
the scrutiny of public spending. Faced 
with the requirements of full and routine 
disclosure, departmental reporting may 
become more balanced and include 
performance that falls below expectations, 
making reports more credible to 
parliamentarians (paragraph 19.65). 


* establish reporting principles and 
stipulate required form and content, based 
on experience to date. This would lead to 
a common basis for assessing the 
performance of departments, using 
consistent terms and interpretations 
(paragraph 19.118). 


* allow central agencies to more 
effectively champion good performance 
reporting (paragraph 19.95). 


¢ confirm the goal of reporting 
performance not only by department but 
also government-wide or by sector 
(paragraph 19.104). 


19.145 However, some pitfalls associated 
with enacting legislation must be 


New Zealand Public Finance Act 1989 recognized. Care needs to be taken to not 
: impose too much rigidity in a legislated 

Lalted Stats reporting regime. Conformance by 

Federal government Government Performance and Results Act, 1993 requirement rather than by conviction 

ae pal is aS ihe Se Act, 1994 would not be very productive. 

regon y 4 ane : . . 

Ty ORS et an Accountability legislation will have the 
se - General ApprOpNaiiens Ace 1022 desired effect only if all concerned — 
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ministers, parliamentarians and officials 
— show their continuing commitment to 
good reporting. 


19.146 A number of other countries 
have recognized the need for legislation, 
as have several Canadian provinces. We 
noted in 1997 that some of these 
jurisdictions had enacted accountability 
legislation; since then, more have done so 
(see Exhibit 19.10). 


19.147 The government should give 
consideration to legislative proposals 
that would require regular reporting of 
financial and non-financial performance 
information to Parliament by 
departments and, as appropriate, by the 
government as a whole, in order to 
demonstrate its commitment to a focus 
on results and to strengthen public 
accountability across government. 


Conclusion 


19.148 Good performance information is 
essential to informing Parliament and the 
public about what the government has 
accomplished with taxpayers’ dollars. It is 
also essential to good management. 
Moving to a results-based culture is not 
easy and we recognize that it takes time to 
establish good performance reporting. 


19.149 Nevertheless, we had expected to 
find noticeable progress by now, and we 
were disappointed. Only marginal 
progress has been made; the majority of 
departments and agencies we examined 
had progressed very little toward good 
performance reporting. This was despite 
the leadership shown by the Treasury 
Board Secretariat. At the present pace, it 
will take too many years for good 
reporting to become routine. 


19.150 Aspects of current reporting to 
Parliament are sound: the focus on 
outcomes, the inclusion of all departments 
and agencies, the efforts made to engage 
parliamentarians, and the perception of 
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reporting as part of a larger government 
priority — managing for results. 


19.151 However, we think significant 
changes are needed in the current 
reporting regime to move things along: 


* the development, communication 
and consistent use of basic reporting ms 
legislation will have 
the desired effect only 


if all concerned show 


* arequirement to report on the 
reliability of performance data; 


* enhanced efforts to link financial and 
non-financial performance information; 


their continuing 
* a stronger internal and external commitment to good 
challenge to the performance expectations 
set by departments; reporting. 


* strengthened review by 
parliamentary committees of reports on 
plans and priorities and departmental 
performance reports; and 


* consideration given to enshrining in 
legislation the requirement for reporting 
performance to Parliament. 


Government’s response: We note that the 
Auditor General agrees with the approach 
the government is taking to improve 
reporting to Parliament, but feels that the 
pace of improvement has been too slow. In 
paragraph 19.151, he summarizes his 
recommendations to increase the pace of 
improvement. 


Other than the recommendation that the 
government actively consider legislation 
as the means for confirming the current 
reporting approach, the government has 
actions under way or planned that respond 
to the Auditor General’s recommenda- 
tions. Results for Canadians: A 
Management Framework for the 
Government of Canada, tabled in 
Parliament in March 2000, provides the 
foundations for the changes in 
management culture that are also essential 
to improve reporting to the public and 
Parliament. Improved Reporting to 
Parliament is one of the six key initiatives 
identified with respect to the 
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implementation of the Results for 
Canadians management framework. 


Among other things, good reporting must 
reflect how departments actually manage 
and deliver results. The federal approach 
puts the emphasis on: 


* results in serving Canadians; 


* measuring performance, making 
improvements and learning from 
experience; and 


¢ being open and transparent with 
Parliament and Canadians. 


Better linking of results and cost 
information and other improvements in 
measuring performance are key elements 
of this plan, as is continuing to work with 
Parliament with the aim of improving 
electronic access and developing 
standards for reporting. 


In addition to its recommendations, the 
chapter notes some of the recent 
achievements in such areas as 
collaborative initiatives and electronic 
reporting. The chapter is balanced, 
recognizing significant accomplishments 
to date, and identifying areas that need 
further improvement. For these reasons, 
we believe the chapter is helpful and can 
serve as a benchmark for measuring 
further progress. 
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ty About the Audit 


Objectives 


One objective of the audit was to assess progress in reporting performance information to Parliament, by 
examining the following: 


¢ the current state (quality) of departmental performance plans and reports; 
© the leadership provided by Treasury Board Secretariat in furthering reporting; and 


¢ the use that standing committees make of performance plans and reports, and any impediments to that 
use. 


We also set out to assess the strengths and weaknesses of the federal regime for reporting on performance. 


Scope 


We reviewed the Estimates documents of 47 departments and agencies, specifically their reports on plans and 
priorities for 1995-96 to 2000-01, and departmental performance reports up to and including 1999. These 
were the larger departments and agencies, including the 16 that had piloted performance reports in 1996, and 
a sample of smaller agencies. We followed up on the observations and recommendations of our 1997 Report, 
Chapter 5, Reporting Performance in the Expenditure Management System. 


As this was a government-wide audit of the current status of performance reporting and its progress, we did 
not audit the reliability or accuracy of performance information in reports on plans and priorities and 
departmental performance reports. 


We reviewed testimony from a number of hearings on Estimates documents held by standing committees. 


Criteria 


We assessed performance reporting against the criteria set out in our 1997 chapter on performance reporting, 
namely: 


* clear presentation of context and strategies; 

* clearly and concretely stated performance expectations; 

* reporting of outcome-focussed performance accomplishments against expectations; 
* demonstrated capacity to learn and adapt; and 

* reporting of fair and reliable information. 

More detailed expectations for good performance reporting are provided in Exhibit 19.2. 
We expected that the Treasury Board Secretariat would: 


* have procedures in place to ensure a reasonable rate of progress in improving performance reporting; 
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® provide adequate guidance and direction, leadership, and demonstrated commitment, including the use of 
incentives and the building of expertise in departments; 


* gather and disseminate best practices and lessons learned; 


¢ work with parliamentarians to develop means of enhancing parliamentary use of departmental 
performance information, and 


¢ report to Parliament on the government-wide progress of performance reporting and managing for results. 


We expected that parliamentarians would identify the use they make of departmental performance reports and 
the impediments to that use. 


In examining the reporting regime, we looked for the extent to which it allows readers of performance reports 
to assess how well an entity is performing, supports accountability to Parliament, and is consistent with other 
government initiatives. 


Approach 


To assess the state of performance reporting, we developed a rating system based on our criteria for good 
reporting. We developed indices for the key criteria of setting out clear and concrete accomplishments and of 
reporting outcome-focussed accomplishments against expectations. We used a similar approach to examine 
the three other criteria of good reporting. We also undertook a more qualitative analysis, reviewing the 
Estimates documents for general readability, presentation, and consistency from year to year. 


We conducted interviews with departmental and Treasury Board Secretariat officials and discussed improved 
reporting with a number of standing committee chairs and vice chairs. We also interviewed Library of 
Parliament staff about the use of performance information by parliamentary standing committees. We 
observed several standing committee hearings and reviewed related testimony and several reports pertaining 
to parliamentary involvement in the review of Estimates documents. 


We reviewed Treasury Board Secretariat documents and attended several events sponsored by the Secretariat 
in support of performance reporting. We reviewed literature on accountability legislation and reporting 
practices in other jurisdictions. 


Audit Team 


Assistant Auditor General: Maria Barrados 
Principal: John Mayne 
Directors: Carol Motuz and Patricia MacDonald 


Christina Brooks 
Adeéle Lamoureux 
Erin Molloy 

Tom Wileman 


For information, please contact John Mayne. 


19-36 Report of the Auditor General of Canada — December 2000 


Appendix A 


Reporting Performance to Parliament: Progress Too Slow 


Departments and Agencies Whose Reports We Examined 


Agriculture and Agri-Food Canada* 
Atlantic Canada Opportunities Agency * 
Atomic Energy Control Board 


Canada Economic Development for the Regions of 
Quebec 


Canadian Centre for Management Development 
Canadian Heritage 

Canadian Human Rights Commission 

Canadian International Development Agency 
Canadian International Trade Tribunal 


Canadian Radio-television and Telecommunications 
Commission 


Citizenship and Immigration Canada 
Competition Tribunal 

Correctional Service Canada 
Department of Finance 


Department of Foreign Affairs and International 
Trade 


Department of Justice 

Environment Canada* 

Fisheries and Oceans * 

Health Canada 

Human Resources Development Canada 
Immigration and Refugee Board* 


Indian and Northern Affairs Canada and Canadian 
Polar Commission* 


Industry Canada 

National Defence 

National Energy Board* 
National Film Board 
National Library of Canada 
National Parole Board* 
Natural Resources Canada* 


Natural Sciences and Engineering Research Council 
of Canada 


* Departments and agencies that piloted performance reports in 1996. 


Office of the Commissioner of Official Languages 
Office of the Correctional Investigator 
Office of the Superintendent of Financial Institutions 


Offices of the Information and Privacy 
Commissioners 


Public Service Commission of Canada 
Public Works and Government Services Canada 
RCMP Public Complaints Commission 


Revenue Canada (now Canada Customs and 
Revenue Agency) * 


Royal Canadian Mounted Police* 


Social Sciences and Humanities Research Council of 
Canada 


Statistics Canada* 

Status of Women Canada* 
Transport Canada* 
Treasury Board Secretariat* 
Veterans Affairs Canada* 


Western Economic Diversification Canada 
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Appendix B 


Setting Outcome Expectations 


Clear performance expectations allow a reader to understand the result that is desired. Concrete performance 
expectations indicate not only the direction sought but also “how much” of the result is sought. Without this indication 
of “how much”, the reader has no way of assessing whether the accomplishments reported are adequate, show 
progress, or meet expectations. 


Making expectations concrete does not necessarily mean expressing them as numerical targets. Many performance 
expectations can and should be expressed in this way. But there is a danger that emphasizing only numerical or 
quantitative targets may limit the stated expectations — and hence reporting — to what can readily be quantified. 
More important, results could be neglected or their achievement undermined by focussing on results that are less 
important but easier to measure. 


In addition to numerical targets, concrete performance expectations may be expressed as: 


* anumerical range rather than a specific amount; 

* the existence or lack of a result (0-1 variable); 

¢ a defined scale of accomplishments, such as “good”, “fair” and “poor”; and 
* asignificant contribution to a result, where “significant” has been defined. 


There is a “results chain” of outputs, intermediate outcomes and end outcomes that describe a program. This chain is 
what is expected to occur, and departments ought to be able to specify in measurable terms the magnitude and timing 
of at least some of these results. With this in mind, concrete expectations can mean that at least some of the specific 
results in the chain are measurable. Clear expectations would then include the clarity of the logic behind the linkages 
between different levels of results. 


We can then look for a clear and concrete chain of results rather than of necessarily only discrete expected results. 
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Status of Our 1997 Recommendations on Performance Reporting 


Recommendations 


Comments 


Departments should: 


a: 


| 


Ensure that their reports contain appropriate references to more 
| detailed sources of information (1997 Chapter 5, 
| paragraph 5.46). 


Satisfactory progress. A number of departments give Web site 
addresses for more information. 


“Include in their performance reporting reference to related 
activities elsewhere in government, as well as discussion of all 
the key instruments that they use to achieve their objectives, 
including those instruments that are not reflected in estimates of 

_ spending for the coming year (5.59). 


Unknown. This recommendation was not examined in the 
audit. 


Establish clear and concrete statements of the performance 

expected from their lines of business. These should be included 

in their business plans, reports on plans and priorities, and 
performance reports (5.74). 


+4 


Unsatisfactory progress. While statements of commitments 
and expected results have become more clear, they are seldom 
concrete. 


| 
| Have performance reports which are more strategic, providing 


better account of the key aspects of performance (5.84). 


Unsatisfactory progress. Departments are often not selective 
in what they report, overwhelming the reader with activity and 
output information. 


Describe how the activities of their programs contribute to the 
| reported outcomes (5.89). 
| 


Unsatisfactory progress. Few departments provide analyses 
on how their activities and outputs contribute to an outcome. 


Report, where performance does not meet expectations, the 
| performance gap, the reasons for it and the changes they are 
| making to close the gap (5.98). 


} 


“+ 


Unsatisfactory progress. Only a few departments report 
performance gaps, and fewer identify reasons or remedial 
actions. 


Provide, where appropriate, an indication of the strengths and 
weaknesses of reported information and their implications for 
the reported performance. They should develop and implement 
procedures to ensure credibility and accuracy of the 
information reported (5.103). 


Unsatisfactory progress. Performance reports do not 
adequately discuss data or information limitations. Accuracy of 
information was not assessed, but internal audit policy is now 
being revised to include provision of assurance. 


The Treasury Board Secretariat should: 


Ensure the consistency of advice and feedback provided to 
departments on performance reporting (5.114). 


Satisfactory progress. The Treasury Board Secretariat is 
co-ordinating advice and feedback, but definitions and usage 
are inconsistent. 


Strengthen its efforts to document and communicate good 

practices by departments in articulating performance 
expectations, measuring results, and using the information to 
improve programs and report accomplishments (5.122). 


Satisfactory progress. The Treasury Board Secretariat has 
instigated numerous activities to document and communicate 
good practice. 
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Recommendations 


Comments | 


Ensure that individual departments are aware of Treasury Board 
Secretariat use of performance information provided in the 
departments’ business plans and performance reports (5.125). 


Unsatisfactory progress. The Treasury Board Secretariat has 
made little use of results information from departments. 


The government, with Parliament’s agreement, 
should: 


Ensure that the improvements in the reporting regime 
implemented on a pilot basis are made a permanent and formal 
feature of the Expenditure Management System covering all 
departments and agencies, and seek parliamentary approval for 
incorporating these improvements in the business of supply 
(5.53). 


Satisfactory progress. Reporting by all government entities 
continues and commitment to report has been reinforced by 
government. Discussions with parliamentarians on ways to 
improve the regime have been held. Parliamentary committees 
have been engaged on the issue. 


Standing committees may wish to: 


Consider strengthening their review, challenge and use of 
performance plans and reports from departments. Government 
should work with Parliament to develop means for enhanced 
parliamentary use of departmental performance information as 
input to the budget consultations (5.142). 


Little change. There is currently little review of or challenge to 
performance plans or reports, and there is little use of 
performance reports by standing committees. 


Key: 


Fully addressed — The original audit finding has been fully addressed and no further action is needed. Our Office will not follow up. 
Satisfactory progress — Reasonable progress has been made in addressing the original finding, but some additional action is required. 
Our Office will follow up. 

Unsatisfactory progress — Reasonable progress has not been made in addressing the original finding and considerable further action is 
required. Our Office will follow up. 


Source: Report of the Auditor General of Canada, Chapter 5, Reporting Performance in the Expenditure Management System, 1997. 
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Summary of Recommendations of the 1998 Standing Committee on 
Procedure and House Affairs, and the Government’s Response 


mo Areas of Focus Recommendations Response 
| Strengthen the parliamentary review of the | Proposed the creation of a standing Recognized the merit of recommendation 
Estimates and the supply process committee on the Estimates. but cited significant obstacles, including 
policy and budgetary objectives, and 
| workload restraints of current committee 
system. 
Suggested that House of Commons 
leaders consider whether an existing 
committee could be given ongoing 
responsibility for ensuring the 
effectiveness of the supply process. 
Resource reallocation Changes to Standing Orders to allow Stated that previous changes to the 
standing committees studying the Standing Orders initiating the creation of 
Estimates to propose reallocations of up to | the performance reports and reports on 
five percent. plans and priorities had already increased 
standing committee reallocation 
provisions. Anything beyond this might 
raise fundamental constitutional 
| challenges. 
Reporting to Parliament, review and Improve quality of information supplied Agreed with objectives and proposed 
accountability to Parliament on government launching a new initiative, building on the | 
expenditures. success of the existing Improved | 
: Reporting to Parliament Project. Result 
Improve program and legislative review. was the Improved Reporting to Parliament | 
Strengthen government accountability on | Project (Phase 2). | 
Estimates matters. 
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Summary of Proposals for Improved Reporting to Parliament Project 
(Phase 2) and Subcommittee Recommendations 


Areas of Focus 


Government Proposals 


Recommendations of the 
Subcommittee on Procedure and 
House Affairs (June 2000) 


Tailoring Estimates information for 
Parliament 


Departmental reporting on plans and 
priorities and on performance to be based 
on key results and government priorities. 


Performance evidence on results, not 
processes. 


Reporting to cover a three-year planning 
period. 


Establishment of government-wide 
standards for reporting costs. 


Proposed requirements should apply to all 
government departments and entities. 


Estimates should contain full disclosure of 
all relevant information concerning 
activities, programs or services. 


Reporting should cover a three-year 
period (except for large, non-recurring, 
capital projects). 


Streamlining and consolidation reporting 
for Parliament 


User-friendly electronic access, e.g., make 
pilot Estimates, performance and planning 
information sites available on Internet. 


Consolidation of departmental reporting. 


Provide executive summaries for longer 
Estimates reports. 


Should explore possibility of 
consolidation of the reporting of activities, 
programs or services that span more than 
one department into a single heading. 


Reviewing and strengthening 
parliamentary review processes 


Strengthen role of standing committees in 
the Estimates reporting process by 
encouraging departments to engage them 
in reviewing their key results 
commitments. 


Encourage standing committees to review 
reports on plans and priorities and 
performance reports. 


House of Commons should reconsider the 
establishment of a standing committee on 
the Estimates with a mandate to monitor 
and review the Estimates and the supply 
process. 


House of Commons should reconsider 
providing more resources to standing 
committees in order to support the review 
and analysis of Estimates and performance 
reports. 


Improving information to Parliament on 
the government’s expenditure plans and 
the use of funds in relation to these plans 


Explore the merit and feasibility of 
integrating the principles of accrual 
accounting in the budgeting process and 
spending information in the Estimates. 


Establish clear links between information 
contained in the Estimates and the overall 
spending plans contained in the budgets. 


Review of the Standing Orders to identify 
the items that would need to be changed in 
order to implement full accrual 
appropriations. 


House of Commons should consider ways 
of improving parliamentarians’ level of 
understanding, knowledge, and ability to 
use financial, budgeting and supply 
information under full accrual basis. 
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practices embrace the standards recommended by the Canadian Institute of Chartered 
Accountants. The numbered paragraphs in bold face represent recommendations. 
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Main Points 


20.1 In our examination of five large federal departments and 10 programs, we found that several departments 
have made tangible efforts to focus on results. There was evidence of commitment and support by senior 
management, extensive planning activities, identification of key performance indicators and some measuring 
against them. 


20.2 However, we found that only 4 of the 10 programs have information on results that is used routinely. In 
most departments we examined, the persistent state of planning for results — rather than actually managing for 
results — has meant that progress in using information on results has been limited since we reported on this issue 
in 1997. It is time for departments to move beyond planning. There is a need for more widespread measurement of 
the results they accomplish and use of that information to improve programs. 


20.3 The government needs to ensure that the evaluation function plays a more central role in its move toward 
managing for results. The evaluation capacity in many departments has atrophied at a time when more effort, not 
less, is needed to measure the results of programs. 


20.4 More and more, the government needs to manage initiatives that span two or more federal departments. 
“Horizontal” issues like the Federal Disability Agenda are often key priorities of government. Our study of three 
horizontal issues found that, like many departmental programs we audited, they were not being managed for 
results. 


20.5 Over the past several years, the government has been clear about the importance of making results central 
to good management and accountability, and this vision is increasingly being accepted across government. 
However, bringing that recognition into practice will require central agencies to play a stronger role. They need to 
provide encouragement and support for departments to move beyond merely planning for results. We found the 
Treasury Board Secretariat’s leadership too limited and its support too dispersed to be of real help to departments 
trying to manage for results. Nor does the Secretariat have a strategy for dealing with horizontal issues. It needs a 
strategic approach to these issues, given their growing presence in government. 


Background and other observations 


20.6 Managing for results allows managers to make changes once they know what is working and what is not. 
It represents a significant difference in the way government programs are managed. Managers can then pay more 
attention to finding out whether programs are meeting their objectives and less to only carrying out activities or 
setting up structures and processes. 


20.7.‘ The concept of managing for results is now widely accepted across government. As a first step in putting 
it into practice, most departments had to realign their management structures and processes according to the 
results they were seeking, and not to their programs or activities. Each department had to establish 
department-wide objectives and think through how it wanted to structure itself in terms of the results it expected. 
Most of the departments we audited were engaged in significant efforts to plan how they would manage for 
results. 


20.8 In managing a horizontal issue for results, it is critical to have a co-ordinating function that is supported 
by senior management (particularly in a lead department) and that has enough resources to do the job. 
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20.9 Departments receive limited practical guidance on managing for results, and there is little sharing of 
experience. They need more encouragement to move beyond planning for results; there is also a need to identify 
champions. 


20.10 While managing for results presents challenges, there are known, practical ways to deal with them. They 
require that people embrace the idea of learning from experience — which includes learning from mistakes. It 
involves bringing more rigour to understanding past experiences and then adjusting practices in light of this 
learning. 


The response of the Treasury Board Secretariat, on behalf of the government and the departments audited, 
is included at the end of this chapter. The Secretariat accepts our recommendations and indicates the actions 
that it is taking or intends to take to address them. 
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Managing Departments for Results: Not There Yet 


Introduction 


20.11 Canadians continue to expect 
government to manage tax dollars 
efficiently and to ensure that they are 
getting value for their taxes. To manage 
public resources effectively toward an 
intended result, government officials need 
to have credible information on the 
performance of the programs and services 
they manage. This is the information they 
will use to determine whether the results 
they expect are being achieved, and 
whether their programs are working well 
or need to be modified. 


20.12 The government needs similar 
information to account for the funds and 
authorities it has received from 
Parliament. Chapter 19 of this Report 
discusses the progress the government is 
making toward better reporting of 
performance information to Parliament. 


Government Has Committed to a 
Long-Term Focus on Results 


20.13 As early as 1976, the Treasury 
Board Secretariat promoted performance 
measurement as a basis for planning, 
decision making and improving 
management practices. During the 1980s, 
program evaluation was given high 
prominence as a management tool to 
improve program performance. In 1995, 
the government launched the current 
move toward a results-based approach to 
management. The President of the 
Treasury Board announced in his first 
annual report to Parliament, Strengthening 
Government Review (1995), that the 
government was committed to delivering 
programs that work, by moving from a 
focus on rules and processes to a focus on 
results. He said that government wants to 
create a management culture that is 
fact-based, results-oriented, open and 
accountable. To deliver more efficient and 
more affordable programs and services, 


the government has embraced the 
principles of service quality, managing for 
results and continuous learning. 


20.14 The President of the Treasury 
Board has reported annually to Parliament 
on the government’s progress toward 
managing for results. The government has 
reiterated its commitment to a public 
service focussed on results in the Treasury 
Board Secretariat’s publication, Results 
for Canadians: A Management 
Framework for the Government of Canada 
2000. 


20.15 In that publication, the 
Secretariat confirmed that it has a key role 
of leadership and facilitation to assist 
departments in achieving desired results. 
It aims to provide “whole-of-government” 
advice to ensure that Parliament and 
Canadians have the information they need 
to hold the government to account. These 
efforts should promote greater 
responsiveness and a focus on results in 
government operations. 


20.16 The government has undertaken a 
number of reforms since the launch of the 
“managing for results” initiative in 1995. 
These include the reform of Estimates 
documents, the Modernization of 
Comptrollership Initiative, the Financial 
Information Strategy (FIS), an increased 
emphasis on service quality, and 
guidelines on preparing Treasury Board 
submissions that support managing for 
results. All of these emphasize and 
encourage a focus on results. 


What Is “Managing for Results”? 


20.17 Despite the government’s 
commitment to managing for results, it 
continues to be a challenge and the 
concept still is not always understood. In 
our 1997 Report, Chapter 11, we discussed 
the government’s efforts to move toward 
managing for results: 


In the past, managers were primarily 
held accountable for the prudent use 


Government managers 
need to have credible 
information on their 
programs to determine 
if they are working well 
or need to be modified. 


The government has 
undertaken a number 
of reforms since the 
launch of the 
“managing for results” 
initiative in 1995. 
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Managing for results in 
departments requires 
long-term strategic 
planning toward 
objectives while 
operating day-to-day 
activities and systems. 


Making decisions 
based on empirical 
information — learning 
— is the heart of 
managing for results. 


of the resources they were given, the 
authorities they used and the activities 
they carried out.... This did not 
encourage a focus on results produced 
by the resources; rather, compliance 
dominated managers’ attention 
(paragraph 11.9). 


When managing for results, ministers, 
senior officials, and managers... 
make decisions based on what a 
program is achieving for Canadians 
— the results that citizens value — 
and at what cost....Holding managers 
accountable for results encourages 
them to focus more on results. 
(11.7-11.9) 


20.18 A variety of terms are used to 
describe managing for results, including 
results-based management — frequently 
used in government publications — and 
performance-based management. These 
terms are essentially interchangeable, but 
for consistency we will use managing for 
results. 


20.19 Managing public sector programs 
and organizations is a complex endeavour. 
There are several objectives to achieve, 
many information requirements to meet, 
external factors to take into account, 
public scrutiny to withstand, and many 
unknowns to cope with. Managing 
requires long-term strategic planning 
toward stated objectives while operating 
complex systems and processes day to 
day. Managers need to step back from 
time to time, and review activities and the 
outputs of activities to see if they are still 
the most appropriate to reach the 
long-term objectives. At the same time, as 
stated in Results for Canadians, managers 
must also be able to “ensure fairness, 
equity, and reasonableness of treatment to 
protect the broad interests of citizens.” 
These enduring public sector values are 
part of the results governments seek to 
achieve. Exhibit 20.1 illustrates the main 
elements of managing an organization for 
results. 


20.20 The exhibit illustrates the 
traditional cycle of planning, 
implementing, monitoring, reporting, and 
revising plans, but it adds a key 
component, learning: making decisions 
based on empirical information. This is at 
the heart of managing for results. By 
monitoring the results of its activities 
through a variety of means (evaluations, 
audits, performance measures, scans), an 
organization can learn what is working 
and what is not working. These activities 
can also provide information about other 
factors outside its control. 


20.21 For example, monitoring will 
indicate whether AIDS information 
campaigns are helping to change 
behaviour; whether agri-food research and 
development activities are generating a 
positive return on investment (crops with 
higher yield and quality) while conserving 
the environment; and whether 
conservation programs are sufficiently 
reducing the risks to endangered species. 


20.22 Based on information gathered 
through monitoring, the organization can 
then either modify its current operations 
or, more fundamentally, revise its business 
and strategic objectives. The organization 
can report to external bodies such as 
Parliament on what it has accomplished. 
Managers of government programs need 
information from the same sources if they 
are to manage for results and gauge their 
progress toward stated objectives. 
Gathering and assessing the relevant data 
and information — the monitoring 
element in Exhibit 20.1 — requires 
systematic effort and a variety of 
monitoring tools. 


20.23 Acritical step in moving toward 
managing for results is to align a 
department’s various strategic objectives 
with the major groupings of its programs 
and services (often called business lines) 
and then with its individual programs. It is 
essential that departments clearly 
understand how the various program and 
service delivery activities support, and 
will lead to, the achievement of their 
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higher-level or strategic objectives. Many 
departments find this a daunting 
challenge. 


An overview of results and performance 
concepts 


20.24 Discussions of performance and 
managing for results can be confusing, as 
they involve a number of specific terms 
and several concepts that are important to 
distinguish. Exhibit 20.2 presents some of 
the key terms and concepts. 


What We Found in 1997 


20.25 We reported in 1997 that 
managing for results was not widespread 
in government at that time. There were a 
number of government initiatives that 
supported a focus on results, but concerted 
effort was needed to bring about real 
change. We identified a few cases where 
managers were measuring results, 
communicating information on results and 


Performance 
Report 


~ ESTIMATES 


2, 


External 
Factors 


Monitoring Tools: 


¢ Evaluations 

¢ Internal audits 

¢ Environmental scans 
¢ Risk analysis 


¢ Performance measurement 
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using that information to improve results. 
However, across government the practice 
of managing for results appeared quite 
limited and we wanted to encourage its 
progress. We identified lessons learned, 
among them key factors that had 
supported managing for results; they 
included the leadership and commitment 
of senior management and an 
organizational climate that encouraged 
managing for results. We concluded that 
ongoing attention was needed across the 
government if managing for results was to 
be adopted throughout government. 


Focus of the audit 


20.26 Our current audit looked at the 
progress a number of major departments 
have made in managing for results. The 
audit was in part a follow-up on the cases 
we reported in 1997, Chapter 11. This 
time our objectives were to: 


* examine the extent to which selected 
departments are managing for results, 


In 1997, we reported 
that managing for 
results was not 
widespread in 
government and 
concerted effort was 
needed to bring about 
real change. 


Exhibit 20.1 


Managing for Results: A Learning Process 


Strategic 
plans 
Strategic 
Planning 
al 


Bring 


adapting 


a 


| Business | 


plans | 


es sae 
" Business 
adapting 


| 


ESTIMATES 


Report on 
Plans and 
Priorities 


Implementing 
Programs 


Work plans 


| Mea 
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Exhibit 20.2 


Results and Performance 
Terms 


including their use of information on 
performance to manage programs; 


- identify lessons learned in managing 
for results; and 


- assess the leadership the Treasury 
Board Secretariat has provided to support 
managing for results throughout 
government. 


20.27. During this audit, we examined 
five large departments, covering a variety 
of program types. In each department we 
asked officials to help identify areas 
where the use of performance information 
could be expected. Exhibit 20.3 lists the 
programs and business lines that we 
selected (see Appendix A for background 
descriptions). We examined them to 
determine to what extent performance 
information is actually used and to what 


e Are the expected results accomplished? 


¢ Are there undue, unintended consequences? 


The concept of performance deals with how well things are done: 


e Are they accomplished within budget and in the most efficient manner? 


It also deals with whether the performance will continue or improve: 
e Is the organization learning from past experience and adapting? 


Performance, then, covers a number of ideas. Determining the specific aspect of performance to 
measure, and when, is not always straightforward. Government programs undertake a number of 


activities that produce a variety of 


results. Programs deliver two kinds Elder Abuse Prevention Program 
of results: outputs, the direct 
products and services produced by _ | Results Specific Program Result 
overnment activities, and : é 
Z Outputs Public education 
outcomes, the consequences of 
those outputs on Canadians and our ¢ Educational material on elder abuse 
society. Outputs are results that with targeted distribution 
managers can control, while ¢ Anti-abuse campaigns 
outcomes are influenced by factors 
outside the programs of managers. Intermediate Increased awareness 
eas Guconis ¢ Increased co-ordination among 
Government organizations and ; se ; 
; : professional agencies, including 
programs seek to achieve certain : : 
‘ Age AES S police, hospitals and banks 
aims or objectives. Objective is c 
e ¢ 
used to refer to general statements LEO DOR SE SEO! 
of intent, akin to the end outcome eee 
statements. It is the intended or ¢ Improved identification of victims 
planned outputs and intermediate and abusers 
outcomes that are key to managing * Increased number of reported 
for results. Actual performance incidences of elder abuse 
can be assessed only in ¢ Increased number of shelter 
comparison with some level of networks 
expected performance. 
Establishing meaningful End Increase in well-being of older 
performance expectations is outcomes Canadians 
therefore an essential part of ¢ Fewer incidences of elder abuse 
assessing a program’s performance ° Reduced costs associated with 
and managing toward those ends. abuse 
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extent other elements of managing for 
results are present. We also sought to 
identify the lessons these departments 
have learned in managing for results, as 
well as the obstacles they have faced. 


20.28 As follow-up to our 1996 Report, 
Chapter 3, on federal program evaluation, 
we also examined how the same five 
departments plan for evaluation and use 
evaluation results to improve programs. 


20.29 The second part of this chapter 
describes a study that focussed on 
managing horizontal issues for results. We 
identified the challenges that horizontal 
issues present and some factors that have 
helped (and hindered) in managing them 
for, and reporting, results. 


20.30 Further details on our objectives, 
criteria and approach can be found at the 
end of the chapter in About the Audit. 


Observations and 
Recommendations 


What Happened to Our 1997 
Cases? 


20.31 We followed up on the cases 
discussed in our 1997 Report, Chapter 11 
namely, the North American Waterfowl 
Management Plan (Environment Canada), 
Investigation and Control (Human 
Resources Development Canada), 
Canadian Centre for Mineral and Energy 
Technology (Natural Resources Canada), 
and the Travellers Program (Canada 
Customs and Revenue Agency). Our key 
finding was that these programs have 
continued since 1997 to manage for 
results. Appendix B provides further 
details on our follow-up. 


20.32 Our 1997 Report emphasized the 
importance of fostering an approach of 
managing for results in an organization, 
and the important part that senior 
leadership plays in this. Over time, 
programs change and so do leaders. 
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Indeed, each of our case studies has 
changed since 1997, some considerably, 
and all of them have seen a change of 
leadership. Our follow-up observations 
suggest that if implanted well enough in 
an organization, managing for results can 
withstand change and continue, even as 
programs adapt to new circumstances. 


This audit examined 
the progress five large 
departments have 
made in managing for 


A Framework for results. 


Department-Wide Managing for 
Results 


20.33 Based on a review of the relevant 
literature and on the several specific cases 
we examined in the federal government, 
we presented a framework in 1997 for 
successfully managing a program for 
results. The framework had four elements: 


¢ fostering an organizational climate 
that encourages managing for results; 


* agreeing on expected results; 


* measuring actual results to improve 
performance; and 


* effectively reporting performance. 


20.34 We think this framework is still 
valid to use in managing programs for 
results. However, based on what we found 
in our current examination of 


Agriculture and Agri-Food Canada Exhibit 20.3 


e Prairie Farm Rehabilitation 


Administration Programs and Business 


¢ Net Income Stabilization Account Lines Examined 


Correctional Service Canada 
¢ Reintegration of Offenders 
e Aboriginal Reintegration 
Environment Canada 

¢ Nature 

¢ Canadian Wildlife Service 
Industry Canada 

¢ Trade 

¢ Connectedness 


Natural Resources Canada 
* Canada’s Model Forest Program 


¢ Aeronautical and Technical Services 
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department-wide efforts, we have 
modified the framework slightly. We have 
highlighted the need to manage for results 
department-wide. We have also pointed to 
the need to use performance information 
for improving programs and identifying 
good practices (see Exhibit 20.4). 


framework. Reaching agreement on the 
broad grouping of results that a 
department is trying to achieve is an 
important step in organizing to manage for 
results. A department needs its key players 
to agree on the strategic results it intends 
to achieve, on the general strategies it will 
use, and on the way it will structure itself 
to achieve those results. We observed that 


20.35 In addition, we have highlighted 


the need to develop and reach agreement 
in the department on a strategic results 


departments are using their business line 
structure — the planning, reporting and 
accountability structure — to link the 


Exhibit 20.4 


Framework for Managing Departments for Results 


Managing an organization for results requires: 


1. Fostering an organizational climate that encourages managing for results 


Fostering an appropriate organizational climate is essential and requires: 


@ 


Demonstrating commitment and leadership. Commitment is demonstrated by such things as strong senior support, widely 
communicated mission statements, effective guidance to line managers and thinking that extends over a number of years. 


Using external and internal levers. Interests in common with external partners are recognized and used along with external and 
internal pressures and events to encourage a focus on results. 


Creating supporting incentives. Appropriate incentives are in place to encourage the measurement of performance and the use 
of performance information. 


Developing a capacity to learn and adapt. Learning from past experience, proactive monitoring of the environment and 
developing a capacity to cope with changing circumstances are encouraged. 


Building expertise. Learning is supported through training, professional development, use of a common terminology and the 
sharing of experience. 


2. Agreeing on a strategic results framework 


The organization needs to come to an agreement among key players on the overall strategic results it intends to achieve and how 
best to structure itself to achieve them. 


Agreeing on strategic objectives and strategies. Agreement is reached on a set of strategic objectives for the organization, 
outlining what it intends to accomplish using which strategies over a multi-year time frame, in light of the organization’s 
mission, past experience and external environment. 


Agreeing on results alignment. Agreement is reached on how the strategic objectives are aligned down through the 
organization to individual program areas. 


3. Agreeing on expected results and strategies for programs 


Managing for results entails reaching reasonable agreement among departmental management, program staff, external partners and 
other key stakeholders on what outcomes the programs are intended to accomplish and the specific strategies for achieving them. 


Agreeing on outcome objectives and strategies. Agreement is reached on a set of outcome objectives for each program and the 
strategies to be used, outlining the program’s intended accomplishments in light of the organization’s strategic objectives. 


Agreeing on performance expectations. Agreement is reached on a set of realistic and challenging statements of what is 
intended to be accomplished. Expectations are clear and concrete, relate to the outcome objectives and specify a time frame for 
their achievement. 


Agreeing on performance indicators. Agreement is reached on a manageable set of qualitative or quantitative indicators of 


each program’s performance that relate to the agreed outcome objectives, and that will be used to assess and manage the 
program. 
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Exhibit 20.4 (cont'd) 


4. Measuring results to track performance 
| 
The organization needs to gather and communicate information on performance: 


* Measuring performance. Practices exist for collecting and assessing program performance and costs using the indicators 
identified and agreed on. 


¢ Evaluating programs. Practices exist to measure the performance of programs through evaluation. 


¢ Reviewing and updating indicators and measures. Based on experience gained, changing circumstances and experience in 
similar programs elsewhere, the performance indicators, expectations, and measurement and communication strategies are 
periodically reviewed and improved. 


5. Using information to improve performance 
Realizing the benefits from managing for results requires: 


* Using performance information. The performance information gathered and assessed is used to improve program performance. 


¢ Identifying good practices. Based on what is found to work and not work, lessons and good practices are identified and used to 
improve performance. Learning occurs. 


6. Effectively reporting performance 


Effective reporting to allow interested stakeholders to judge how well programs and the organization are doing requires: 


* Describing the context and the strategies used. The mission, mandate and objectives of the organization, the major strategies 


used, and the related external context are discussed. 
¢ Stating meaningful performance expectations. Clear and concrete key performance expectations are focussed on outcomes. 


| * Reporting performance accomplishments against expectations. Key accomplishments are related to expectations and are 
attributed, in whole or in part, to the activities undertaken. 


* Demonstrating the capacity to learn and adapt. The ability to learn from past performance and to adapt the program to 
external changes is demonstrated. 


¢ Reporting on the fairness and reliability of the performance information. The performance information reported is fair and 
reliable. 


department’s internal management and 
accountability regime to its objectives, 
business lines, resource requirements and 
performance targets. 


Managing for Results Still Not an 
Integral Part of Managing 
Departments 


What we expected 


20.36 In the three years since our 1997 
study, the government has remained 
committed to a focus on results. In 1997, 
we suggested it could take the government 
four to five years to shift its focus from 
managing activities to using performance 
information routinely for better 
management of programs. 


20.37 This time in our audit, we 
expected to find that the practice of 
managing for results had advanced, with 
its key elements evident in many areas of 
government. We looked for signs of senior 
support, performance expectations that 
were well articulated and understood, 
performance indicators, and systems for 
collecting and measuring data. We 
particularly expected to see performance 
being monitored in many government 
programs and the information being used 
to improve the management of operations 
and policy. 


More attention to results 


20.38 Overall, we saw tangible efforts 
in all the audited departments to foster a 
climate of results, including the strong 
commitment and support of senior 
management. Most of the departments 
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Performance 
measurement and use 
of performance 
information has made 
limited progress 
since 1997. 


A department needs 
its key players to 
agree on the intended 
results and how best 
to structure itself to 
achieve them. 


were engaged in significant efforts to plan 
how they would manage for results. All 
the programs we examined were 
developing or fine-tuning performance 
measures. 


20.39 We reported in 1997 that 
managing for results was not widespread 
across government and had made limited 
progress. Now there is more attention to 
results. All government departments now 
set out their key results commitments, 
which are used as the basis for their 
planning, reporting and accountability 
structure. In the departments we examined 
however, the actual measurement and use 
of performance information has made 
limited progress. 


20.40 Alignment for results is needed. 
As we have indicated earlier, a department 
needs its senior managers to understand 
and agree on its key results commitments 
and how it will structure itself to achieve 
them. During our audit, we have found 
that departments, to focus on results, often 
need to realign their management 
structures and processes according to the 
key results they are seeking and not to 
their programs or activities. This can be 
challenging. 


20.41 The department needs to begin to 
consider how to structure itself in terms of 
its key results commitments. This may 
require changing the organizational 
structure or establishing business lines that 
span parts or all of the organization. 
Departments frequently manage by 
business line, each headed by an assistant 
deputy minister who is responsible for the 
key results commitments that contribute to 
the agreed department-wide strategic 
results framework. 


Use of performance information is 
limited 


20.42 Our audit looked at how 
performance information is used at the 
department-wide or corporate level. We 
found little use of performance 


information to improve management, or to 
shift priorities and reallocate resources 
across business lines. Business lines in the 
five departments we examined are quite 
large and involve sizable groupings of 
departmental activities and programs. 
Perhaps as a result, these business lines 
are often managed separately by the 
deputy minister and the responsible 
assistant deputy minister. 


20.43 In two of the five departments 
(Agriculture and Agri-Food Canada and 
Correctional Service Canada), we found 
that corporate managers receive 
information regularly on a number of 
performance indicators. This has allowed 
the departments to track important trends 
in performance and to keep informed of 
key changes in their environment. For 
example, Agriculture and Agri-Food 
Canada uses a “dashboard” of 
performance indicators to identify 
emerging problems and to signal the 
possible need for more information or 
future decisions. 


20.44 We also looked at how 
performance information is used by 
programs. We asked the five departments 
to identify the programs most likely to be 
managing for results and, in particular, 
most likely to be using performance 
information. It took some time before 
most of the departments could identify 
such programs, often with our help. It was 
clear that managing for results was not 
widespread in most of these departments’ 
programs. 


20.45 In October this year, we reported 
our assessment of financial management 
capabilities in selected departments. A 
common gap we found again was the 
limited ability to integrate financial and 
non-financial information. Most 
departments indicated that they had only 
begun the task of integrating financial and 
operational systems. In this audit, we also 
found a poor link between results and cost 
information. In our examination of 

10 programs, we found that departmental 
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managers had only limited cost 
information available. 


Planning for Results But Not 
Enough Implementation 


Extensive planning to manage for 
results 


20.46 Developing the tools to use in 
managing for results is an important step. 
We found considerable department-wide 
planning under way in the five 
departments we reviewed. Common 
planning activities included developing a 
performance and/or accountability 
framework, identifying and revising key 
indicators of performance to measure, and 
revising program business plans in light of 
the department’s strategic results 
framework. The following are some 
planning activities we noted: 


- At Agriculture and Agri-Food 
Canada, logic charts were being 
developed to demonstrate the linkages 
among a program’s inputs or activities, its 
outputs, and its outcomes. Exhibit 20.5 
illustrates a logic chart that the Prairie 
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Farm Rehabilitation Administration 
developed for the Water Aeration of 
Surface Reservoirs project. The 

exhibit shows one of three results 
expected for this project and considered 
necessary to achieve the long-term 
benefit: better quality of water in the rural 
Prairies. The logic chart includes clear 
statements of intended outcomes; 
performance measures; and concrete, 
measurable targets with timeframes. 


¢ In Environment Canada, numerous 
activities were under way to implement 
the Department’s performance 
measurement strategy across the business 
lines. Planning activities in the Nature 
business line have included selecting 
performance indicators, defining data 
collection strategies, and building logic 
models for key intended results — 
including “conserving biological 
diversity”, a result delivered primarily by 
the Department’s Canadian Wildlife 
Service. The Department has stated that in 
the coming months the business lines will 
determine what data they need to support 
the logic models and indicators, and then 
will implement data collection strategies 


Planning activities 
included developing 
performance 
frameworks, selecting 
performance 
measures, and 
revising program 
business plans. 


Exhibit 20.5 


Logic Chart for the Water Aeration of Surface Reservoirs Project 


EXPECTED RESULT 


To maintain and enhance water quality in dugouts and reservoirs and 


reduce the need for expensive point-of-use water treatment systems 


Increase Increase 
information base awareness 
1999-2000 2000-01 
Target Targets 
Develop a design * 500 producers and 


representatives of 
agriculture sector who 
received a demo and 
explanation of 
benefits. 


standard for aeration of 
surface water storage 
reservoirs. 


Results 


Preparation of three fact 
sheets that explain 
proper implementation 
methods. 


* Fact sheets distributed 
to individuals. 


Develop 
baseline 
measurement 
information 


=> 


Collect data to 
measure key 
indicators 


Increase 

adoption Long-term 

2003-10 benefit 
Outcome 


Producers are aerating 
dugouts. 


Will be measured by: 


* percentage of new 
dugouts constructed 
with aeration. 


* percentage increase 
in sales of aeration 
equipment. 


Better water quality 
in rural prairies. | 


Source: Agriculture and Agri-Food Canada 
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Departments have 
been slow in moving 
beyond the planning 
stage. 


in a phased approach over the next several 
years. 


« Natural Resources Canada has 
continued to develop its performance 
measurement framework, and has 
focussed on developing indicators of 
performance. The performance 
measurement working group has consulted 
extensively with program managers and 
stakeholders to review draft performance 
indicators and seek guidance on refining 
them. Natural Resources Canada has 
developed criteria to assess and refine 
each indicator and establish targets, where 
appropriate. Many of the current 
performance indicators have targets that 
are “directional” (for example, “to 
maintain or improve Natural Resources 
Canada’s contribution”). The Department 
has stated that after it evaluates the 
directional targets, it will be in a position 
to consider establishing numerical targets. 


A need to move beyond planning 


20.47 As already noted, we found some 
progress in measuring performance, but 
use of the information is still limited. 
However, we found that planning to 
manage for results is extensive, especially 
at the corporate level, and in many cases it 
has gone on for years. Departmental 
officials often told us they expect to use 
performance information more regularly 
across the department, “soon”. Practical 
use of performance information and 
costing of results always seem to be 
promised for “next year”. 


20.48 Commitment to move beyond the 
planning stage needs to be demonstrated 
across the organization. Some departments 
seem to be searching for the “perfect” 
departmental structure to manage for 
results, and they continually change what 
they have set out. In other departments, a 
change of deputy minister or minister 
triggers a reconsideration of the strategic 
results framework they have been 
developing. 


20.49 In 1997, we reported on the good 
beginning of managing for results in 
Environment Canada. It had a strategic 
results framework in place and appeared 
ready to implement it across the 
Department to manage for results. Yet 
three years later, it is essentially at the 
same corporate planning stage, with a 
different results framework in place. It is 
still trying to agree on performance 
measures, and for the most part 
measurement has not yet begun. Industry 
Canada has been developing its strategic 
results framework for two years, with 
numerous iterations. The Department 
agreed on performance measures only 
recently. Natural Resources Canada and 
Agriculture and Agri-Food Canada have 
also continued to revise their planning 
tools, including their results frameworks 
and performance measures. The reasons 
may differ but the result is the same: not 
moving quickly enough beyond the phase 
of designing departmental frameworks 
that are crucial to the next stage of 
managing for results. 


20.50 The challenge for departmental 
managers is to move on — to implement 
the various planning tools, measure 
against the established indicators and use 
this information to improve the 
management of their programs. Managing 
for results means that managers may 
change what they do once they know what 
is working and what is not. This applies 
equally to developing and using 
performance measures. Sound measures 
are those that managers will use over a 
period of time. If measures provide data 
that prove to be poor or not useful, the 
measurement strategy needs to change. 


20.51 Senior management of 
departments should move their 
organizations beyond merely planning 
to manage for results. They should get 
on with measuring performance and 
using the information to achieve the 
intended results. 


20-16 


Report of the Auditor General of Canada — December 2000 


Some Are Managing for Results 


20.52 We did note some elements of 
managing for results in all the selected 
departments. 


Performance information used in some 
business lines and programs 


20.53 In the 10 programs we examined, 
we did find some use of information on 
results (see Exhibit 20.6). We found four 
programs that use performance 
information routinely to improve their 
operation and design: Agriculture and 
Agri-Food Canada’s Net Income 
Stabilization Account; Prairie Farm 
Rehabilitation Administration; 
Correctional Service Canada’s 
Reintegration of Offenders; and Natural 
Resources Canada’s Aeronautical and 
Technical Services. In these programs, the 
commitment of senior and middle 
management is evident; indicators of 
performance are clearly linked to planned 
results; and the information is used in 
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planning and in program design. Managers 
consider performance information to be 
reliable, and useful for decision making. 
For example, information on results has 
helped in identifying the need for new 
programs and redesigning existing 
programs. 


20.54 The six other programs ranged 
from some use of performance 
information to none. They used 
information more to improve program 
operations and design than to revise 
performance expectations and reallocate 
budgets. Overall, the programs we 
reviewed used performance information 
less than we had expected. 


20.55 A previous audit addressed 
managing government services for results. 
Our April 2000 Report, Chapter 1, Service 
Quality cited several services that were 
collecting performance information. With 
the exception of telephone services, 
however, few of the service lines 


| Performance Information Exhibit 20.6 
| Improve operations Update expectations and GeaeePeronnanes 
_ Programs and design reallocate budgets Sah 
— Information in 10 Programs 
| Agriculture and Agri-Food Canada 
¢ Prairie Farm Rehabilitation 
Administration Routine Some 
¢ Net Income Stabilization 
Account Routine Some 
Correctional Service Canada 
¢ Reintegration of Offenders Routine Some 
¢ Aboriginal Reintegration Some Some 
Environment Canada 
¢ Nature Limited None 
* Canadian Wildlife Service Some Some e 
Industry Canada 
e 6Trade Limited None 
¢ Connectedness Some None 
ais! oo i a 
Natural Resources Canada 
* Canada’s Model Forest 
Program Some Limited 
* Aeronautical and Technical 
Services Routine Some 
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Senior commitment, 
leadership and 
appropriate incentives 
are key elements in 
moving forward with 
managing for results. 


examined were using performance 
information to improve service. 


20.56 Our follow-up of the 1997 audit 
of Human Resources Development 
Canada’s transition to managing for 
results has reported that the Department 
has continued to make progress (Auditor 
General’s Report, December 2000). In the 
Employment Insurance program, the 
protection of public funds against fraud 
and abuse continues to be managed for 
results; and key performance information 
is analyzed more thoroughly, making it 
more useful to managers. 


Other elements of managing for results 
are present 


20.57 Although we found that 
information on results was being put to 
only limited practical use, other elements 
of managing for results were evident in 
the programs and business lines we 
examined. Those elements included: 


¢ fostering an organizational climate 
that encourages managing for results; 


* measuring performance; and 


« developing a strategic framework 
that outlines key results commitments and 
planned performance expectations. 


20.58 Fostering an organizational 
climate for results is key. Senior 
commitment and leadership are key 
elements, particularly in moving forward 
with managing for results, past the 
planning stage. Those elements need to be 
supplemented by appropriate incentives 
that support performance measurement 
and the use of performance information. 
More efforts are needed to remove 
impediments and overcome resistance. 


20.59  Insome of the programs we 
examined, we found some form of 
performance contract among the senior 
managers that encourages them to focus 
on results. Performance contracts 
identified the results expected over the 
coming years, and were tied to relevant 
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business plans. Performance contracts are 
a good incentive to manage for results, to 
the extent that they specify results and not 
just activities. 


20.60 We observed that the pay-at-risk 
initiative was used in a few departments to 
focus attention on results and performance 
reporting. This government-wide initiative 
was introduced in 1998 to allow managers 
to earn up to 10 to 20 percent of pay on 
the basis of performance, measured 
against targets agreed to at the beginning 
of the year. 


20.61 We found a number of other 
tangible initiatives to foster a climate that 
supports managing for results. Some 
departments recognized their limited 
capacity and expertise in this area. Some 
of them addressed it by creating a 
performance measurement group; others 
recruited consultants; and some made 
good use of the experience in their 
evaluation unit. 


20.62 We also observed the following 
examples of a supportive environment: 


¢ The Commissioner and senior 
officials of Correctional Service Canada 
were committed to obtaining information 
on results and using the information to 
analyze the results. Evidence of that 
commitment is the regular monitoring of 
results, which has pointed to specific 
programming needs of offenders. For 
example, the Aboriginal reintegration 
program is being developed to meet the 
programming needs of the Aboriginal 
inmate population. There is strong buy-in 
from middle managers, who ensure that 
data systems are maintained; they use 
performance information in managing 
their programs. Finally, the Service uses 
the pay-at-risk initiative as a lever to 
ensure that managers focus on results, and 
it closely monitors their performance 
contracts. 


¢ Jn Agriculture and Agri-Food 
Canada, strong commitment and 
leadership was evident at the levels of 
deputy minister and assistant deputy 
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ministers. The Department has dedicated 
resources, with additional Treasury Board 
funding, to develop a “dashboard” of 
performance indicators that were closely 
monitored by the Deputy Minister. In 
addition, a working group was established 
to launch the Department’s performance 
measurement initiative. The Assistant 
Deputy Minister, Corporate Services 
Branch, provided guidance to the working 
group and has chaired an advisory 
committee on performance measurement. 
The committee members included the 
assistant deputy minister over each 
business line, senior managers, and 
several stakeholders. 


¢ Industry Canada has designated 
assistant deputy ministers as champions of 
its “strategic objectives”, which 
correspond to business lines. The Deputy 
Minister reinforced a strong message to 
managers that resource allocation among 
programs should be based on their ability 
to show results. Dedicated resources were 
given to the Corporate Planning and 
Performance Co-ordination Directorate to 
manage the process of preparing reports 
on plans and priorities and departmental 
performance reports. The Directorate 
represented all sectors of the Department. 
It also had a mandate to develop and 
implement a performance measurement 
strategy and support the corporate vision 
of results. 


20.63 Some progress in measuring 
performance. Managing for results means 
measuring outputs and outcomes with a 
view to learning from experience. We 
found that at present, it is largely activities 
and outputs that are measured. Over time, 
we would expect to see an increase in the 
measuring of intermediate and end 
outcomes that are linked to the program 
activities. In time, we would expect 
departments to develop cost-effective 
measurement tools to cost activities, as a 
basis for allocating resources according to 
expected results. Finally, evaluation can 
help in this measurement task. 


Managing Departments for Results 
and Managing Horizontal Issues for Results 


20.64 The five departments we audited 
set out a strategy for measuring results, 
either in the departmental report on plans 
and priorities or in a separate corporate 
planning document. We observed some 
level of performance measurement in the 


10 programs we audited: 
Performance 


measurement is still 
largely focussed on 
activities and outputs. 


* Correctional Service Canada 
routinely measures its accomplishments 
and results, particularly in its 
Reintegration business line. The Service 
has a vast range of indicators and 
measures that senior managers use to 
manage this business line. Integrated data 
systems track performance at the 
individual level (for example, how each 
offender is progressing through his/her 
release plan), at the institution level 
(make-up of the offender population), and 
at the regional and national levels (rates of 
parole revocation and release rates). 


« Managers of Industry Canada’s 
strategic objective, “Connectedness”, 
recognize that medium- and longer-term 
impacts should be measured. Recently 
developed performance information 
focusses on the socio-economic impacts of 
“connectedness” — people’s ability to use 
information and communication 
technologies and to interact through them. 
Work continues with the Conference 
Board of Canada to further develop a 
“connectedness index”, which is focussed 
on quantifying affordability, access, usage 
and socio-economic indicators. One of the 
next steps will be to refine a framework 
for examining the impacts and outcomes 
of connectedness. 


Over time, we expect 
to see an increase in 
measuring 
intermediate and end 
outcomes that are 


¢ Natural Resources Canada’s linked to the program 


Aeronautical and Technical Services has 
made significant strides in measuring 
performance in this largely 
production-oriented environment. Cost 
recovery of aeronautical services and 
products (navigation charts and 
publications) has meant extensive tracking 
of cost information and a strong client 
focus. The program has implemented 
principles of a quality management 
framework to assure product quality, 


activities. 
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efficiencies, and responsiveness to clients. 
Service standards are set and tools to 
measure client satisfaction are used 
regularly. 


Getting on with 20.65 Getting on with measuring the 
4 results they achieve is perhaps the most 
measuring the results immediate challenge departments face. 
they achieve is the The next step beyond planning to manage 
: ; for results is implementing a measurement 
most immediate strategy — actually measuring a wide 
challenge departments range of outputs and outcomes, learning 
which measures are most useful by using 
the information, and adjusting 
measurement in the future. 


face. 


Learning From Others’ Experience 


20.66 Initiatives in managing for 
results are common in other 
jurisdictions. Our 1997 study noted a 
number of results-oriented initiatives in 
several jurisdictions, both in Canada and 
elsewhere. We observed in 1997 that some 
of these jurisdictions had enacted 
accountability legislation; since then, 
more have done so. 


20.67 In Alberta and British Columbia, 
legislation requires both government-wide 
and ministry preparation of multi-year 
business/performance plans to be 
presented to the legislature. It also 
requires annual reporting of progress 
made toward achieving the expected 
results set out in those plans. In Quebec, 
new legislation focussed on improved 
public service is highly supportive of 
managing for results. In order to enhance 
accountability, the use of multi-year 
strategic plans and performance 
agreements is now mandatory for all 
departments. 


20.68 Chapter 19 of this Report, 
Reporting Performance to Parliament: 
Progress Too Slow, recommends that in 
light of the modest progress made in 
reporting performance to Parliament, it 
may be time for the government to 
consider introducing accountability 
legislation. As we observed in the audit 


discussed here, managing for results is 
still not an integral part of managing 
departments. There is a clear need to get 
beyond planning and secure further 
progress. Although legislation alone will 
not ensure good reporting or managing for 
results in departments, it could serve as a 
signal, as Chapter 19 suggests, that 
measuring and using performance 
information is not a passing fad. 


20.69 Literature review identifies 
lessons learned. We conducted an 
extensive review of the literature to 
identify lessons learned and trends in 
other jurisdictions with experience in 
managing for results. Exhibit 20.7 
presents a summary of these lessons, 
related to promoting favourable conditions 
for implementation; developing 
performance measurement systems; and 
using performance information. A full 
report can be found at 
www.oag-bvg.gc.ca. 


20.70 We noted many consistencies 
between the findings of our literature 
review and findings from our examination 
work in departments on the lessons 
learned in implementing managing for 
results. 


Need for Concerted Leadership by 
Treasury Board Secretariat 


20.71 While the Treasury Board 
Secretariat has actively promoted 
improved reporting in a number of quite 
tangible ways (as discussed in Chapter 
19), its efforts to encourage managing for 
results in the day-to-day operations of 
departments have been less evident. 


20.72 A number of Treasury Board 
initiatives and publications relate to 
managing for results, including the 
Modernization of Comptrollership 
initiative, the President of the Treasury 
Board’s annual report to Parliament on 
Managing for Results, and the Treasury 
Boards publication, Results for 
Canadians: A Management Framework 
for the Government of Canada (see 


20-20 Report of the Auditor General of Canada — December 2000 


Appendix C for a description of these and 
other initiatives). While these provide 
some recognition of the need to manage 
for results, discussion in them of specific 
practices is quite limited. These initiatives 
originated in different parts of the 
Secretariat that give different levels of 
attention to managing for results. It is 
often not their prime focus. 


20.73 Limited evidence that the 
Secretariat uses information on 
departmental results. In 1997, we 
suggested that a significant incentive for 
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departments to manage for results would 
be clear evidence that the Treasury Board 
was indeed using performance information 
in its deliberations. Our interviews at the 
Treasury Board Secretariat suggested that 
while it occasionally uses departmental 
performance information, it does not use it 
systematically or routinely in its decision 
making. Yet this is stated as a “key 
responsibility” of the Secretariat as a 
management board: to “compile 
information sufficient to assess program 
performance and program integrity across 
government.” 


Promoting Favourable Conditions 


Exhibit 20.7 


| Organizational Factors 


¢* A regime customized to manage for results is critical. 


¢ Take time and maintain momentum. 


¢ Linking performance measures to the policy or strategic framework is key. 
¢ Align management systems to support implementation. 


Pilot projects can be useful. 


Human Factors 


¢ Developing a performance management culture is critical. 


* Senior-level leadership and involvement are essential. 

¢ Full participation fosters support for implementation. 

¢ Training and education are key ingredients of success. 

¢ Use existing expertise to support implementation. 

* Communicate the purpose of the performance measurement system 


| * Providing adequate financial and human resources is critical. 
¢ Identifying stewardship over performance measurement process is important. 


A practical understanding of accountability is needed. 


Lessons on Effectively 
Implementing Managing for 
Results 


Developing Performance Measurement Systems 


¢ Use a manageable number of indicators. 
* Clearly define key terms and concepts. 
¢ Using the logic chart can be very helpful. 


¢ Align performance measures with decision-making authority and accountability. 
* Credible performance information is essential. 
¢ Performance standards and targets are essential for measurement. 


¢ Use baseline data to set targets. 


Using Performance Information 


* Demonstrable use of performance information is essential. 
* Evaluation and performance measurement are complimentary. 


* Incentives can be used to foster support. 


* Performance reporting is needed for decision-making. 
¢ Review and adjust performance measurement systems. 


Source: Office of the Auditor General, 
2000, Implementing Results-Based 
Management: Lessons from the 


Literature. 


Report of the Auditor General of Canada — December 2000 


20-21 


—_ 
Managing Departments for Results 
and Managing Horizontal Issues for Results 


The Treasury Board 
Secretariat has a role 
to champion managing 
for results and to 
support departments 
in this effort. 


Evaluation should be 
an integral part of 
program management 
and a department’s 
strategy for managing 
for results. 


20.74 Treasury Board Secretariat has 
not identified “results” expected in 
government’s managing for results. The 
Secretariat has not set out the results 
planned for the government in managing 
for results — specifically, what is 
expected in departments and by when. In 
the absence of clear expectations, it may 
be difficult for the Treasury Board 
Secretariat to know the kind of support it 
should be providing and whether 
departments are making enough progress. 
The Secretariat has not had a clear plan to 
focus its activities in this area. Its 
publication Results for Canadians calls for 
a strong focus on results in managing the 
Government of Canada. Officials told us 
that the Secretariat will use this 
publication as a basis for action plans to 
meet the government’s commitment to 
managing for results. 


20.75 The Treasury Board Secretariat’s 
efforts are present, but diffused. To be 
effective, it needs to concentrate more on 
defining concrete expectations and 
practical strategies for managing for 
results. Otherwise, it is not clear how it 
will fulfil the role it recently set for itself 
as a management board to champion 
managing for results. 


20.76 The Treasury Board Secretariat 
should develop concrete ways to 
champion managing for results in 
government and to support departments 
in this effort. It should actively assemble 
and communicate practices that have 
proved successful in dealing with the 
challenges that departments face. 


Evaluation Is a Vital Element 


20.77 Evaluation has an important role 
to play in managing for results. It can 
provide important information on program 
performance that is not gathered by 
ongoing monitoring systems, and can help 
managers understand why programs are 
working or are not. Evaluation can assist 
in the development of the results-based 
management and accountability 


frameworks the Treasury Board now 
requires, which help managers identify the 
information they need to collect in order 
to demonstrate results. We would 
therefore expect evaluation to be an 
integral part of program management and 
part of a department’s strategy for 
managing for results. 


Following up on our 1996 audit 


20.78 In 1996 we reported on the state 
of evaluation in the federal government 
(1996 Report, Chapter 3). We made 
several recommendations to the Treasury 
Board Secretariat and to deputy heads. In 
the current audit, we followed up on those 
recommendations and looked at how 
evaluation is being used in managing for 
results. 


20.79 Revisiting our 1996 
recommendations. The wording of our 
1996 recommendations reflected the 
circumstances at that time. Since then, 
there have been a number of changes in 
the government’s expenditure 
management system and in the 
Secretariat’s reporting to Parliament. We 
made three recommendations to the 
Treasury Board Secretariat, based on the 
following observations. 


20.80 We had found that the bulk of 
evaluation activity was focussed on 
operational and lower-level issues and 
concerns, and that significant aspects of 
government activity were not being 
evaluated and reported. In our view, 
evaluation needs to play a strategic role in 
government as a whole in addition to its 
role in departments. The government 
needs the capacity to effectively evaluate 
interdepartmental and other horizontal 
issues. We recommended that some 
direction should be provided to the 
evaluation functions in departments on 
overall evaluation priorities so that the 
government is able to evaluate the 
important areas, including horizontal 
issues, of government programming and 
interest (paragraph 3.74). 
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20.81 Parliament has frequently 
expressed an interest in the evaluation 
capability of government and the Public 
Accounts Committee has asked to be kept 
informed about the state of evaluation. We 
recommended that the Secretariat should 
report to Parliament on the state of the 
government’s evaluation capacity 
(paragraph 3.86). 


20.82 It was hard to identify what was 
then called evaluation. The “review” 
function was diffusing effectiveness 
measurement, which was central to 
evaluation. We thought that if review were 
to be encouraged, it should have 
appropriate standards that would differ 
from those used for evaluation. We 
recommended that quality standards be set 
out for evaluation and any other related 
effectiveness measurement activity of 
government (paragraph 3.106). 


20.83 In our view, our 1996 
recommendations are still relevant. 


Our assessment of progress 


20.84 We believe that progress on these 
recommendations over the past four years 
has been unsatisfactory. (Appendix D lists 
the recommendations and summarizes our 
assessment of progress). Recently, the 
government has been putting new efforts 
into evaluation. 


20.85 But the Treasury Board 
Secretariat has not yet set out 
government-wide priorities for evaluation, 
as we recommended in 1996. 


20.86 Although the first report of the 
Treasury Board President to Parliament 
(1995) did include evaluation and review, 
the focus of that annual report has 
changed. Parliament has not been 
informed since 1996 about the relatively 
weak state of evaluation in government, 
despite its clear interest in the subject. 


20.87 The Secretariat has draft policies 
proposed for both internal audit and 

evaluation. The current version makes no 
reference to review as a separate activity, 
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which may simplify measurement. The 
draft evaluation policy contains quality 
standards that, if promulgated, would 
satisfy our recommendation for evaluation 
standards. 


Evaluation function has regressed 


20.88 A need to rebuild the evaluation 
function. A recent study by the Secretariat 
shows that over the past four or five years, 
the evaluation function in the federal 
government has regressed. During most of 
this period, the Secretariat’s attention to 
evaluation was limited. Key study 
findings include the following: 


¢ Evaluations are still seen by some as 
a check on management rather than as a 
tool for management, but this attitude 
could be changing. 


¢ There is room to develop managers’ 
capacity to carry out evaluations of their 
programs for the purpose of organizational 
learning and improved implementation. 


¢ Evaluation frameworks (or 
results-based management and 
accountability frameworks), although 
mandated by policy, have not always been 
created or, if created, not always 
implemented. 


* Program Review led to a serious 
undermining of evaluation capacity across 
the government. 


There is a clear need for concerted effort 
to rebuild the evaluation function. 


20.89 There are signs of progress in 
some areas. In line with our 1996 
recommendations, the Treasury Board 
Secretariat is now identifying ways it can 
support the evaluation function. In its 
publication Results for Canadians, it made 
the following commitment: 


Working with departments and 
agencies, the [Secretariat] will 
develop and implement plans to better 
position and strengthen the program 
evaluation and audit functions within 
the broader effort to implement 


There is a clear need 
for concerted effort to 
rebuild the evaluation 
function. 
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modern comptrollership and 
results-based management across 
government. 


20.90 Further, Treasury Board 
Guidelines for the Preparation of 
Departmental Performance Reports 
discuss the important role evaluation can 
play in reporting. In addition, the recently 
revised Policy on Transfer Payments 
requires that departments and third parties 
prepare a results-based management and 
accountability framework and conduct 
evaluations of their programs and projects. 


20.91 Evaluation in departmental 
business planning. Two recommendations 
in our 1996 audit were addressed to 
deputy ministers: that business plans 
include evaluation findings and identify 
priorities for evaluation, and that plans for 
evaluation be developed as part of the 
business planning process. While 
departments are no longer required to 
submit business plans to the Treasury 
Board Secretariat, they are still expected 
to undertake business planning. The five 
departments we examined have made 
satisfactory progress in this area. 


20.92 We looked at evaluation planning 
practices in our five case departments. 
Their practices varied; each produces 
different types of planning documents 
containing different types of information. 
Key business planning documents are 
developed at the business line level. These 
documents often set out strategies for 
measuring performance, including 
planned evaluations. 


20.93 A review of departmental 
evaluation plans in the five departments 
revealed that all of them group evaluation 
activities by business line or by 
departmental objective. Environment 
Canada’s evaluation plan articulates 
clearly how evaluations support managers’ 
efforts to achieve planned results and how 
they help senior managers meet the 
Department’s strategic objectives. All 
departments said they are continuing to 
work on integrating departmental 


20-24 


priorities in their evaluation plans, and 
increasingly they see evaluation as an 
important component of the department’s 
strategy for meeting its objectives. 


20.94 Some recent management 
interest. In two of the departments, we 
heard that evaluation has remained stable 
over the past few years. In the three other 
departments, evaluation is only now 
showing signs of revival after a few 
difficult years: more staff are being hired, 
and the departments are developing 
strategies to strengthen the professional 
capacities of their audit and evaluation 
personnel. Increasingly, program 
managers are contacting the evaluation 
unit for help in developing evaluation 
frameworks (results-based management 
and accountability frameworks), as 
required in Treasury Board approval 
documents. Heads of evaluation in the 
five departments noted that program 
managers are also showing more interest 
in tools that will help them demonstrate 
results and support them in decision 
making. 


20.95 One of the Secretariat’s 
conclusions in its study of the evaluation 
function is that there is a need to renew 
the mandate of evaluation in the new 
management regime. This is clearly a task 
for the Treasury Board Secretariat to 
undertake. 


20.96 The Treasury Board Secretariat 
should strengthen the evaluation 
function to ensure that it plays a key 
role in the government’s move to 
managing for results. The Secretariat 
should also ensure that the evaluation 
function has the capacity to address 
strategic issues, including horizontal 
issues, that are of concern to 
government as a whole and to 
Parliament. 


Conclusion 


20.97 We set out to identify the extent 
to which managing for results was being 
practised in the five selected departments 
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and 10 programs we examined. We found 
a few programs that use performance 
information routinely. We found that other 
elements of managing for results have 
shown some progress. They include 
initiatives to foster a supportive climate, 
extensive planning activities, and some 
measurement of performance. But a lot 
remains to be done before a strategy for 
managing for results is fully established in 
government. 


20.98 Most departments need to move 
beyond planning to manage for results. 
They need to get on with measuring their 
progress toward at least some of their 
planned results, and using that information 
to improve performance. The persistent 
state of planning in most departments has 
meant that progress has been limited since 
we reported on this issue in 1997. 


20.99 We appreciate the many 
challenges that departments encounter in 
their efforts to manage for results, and we 
recognize that this process takes time. In 
our audits and in our review of other 
jurisdictions that are managing for results, 
we identified many of these challenges — 
in particular, the challenge of fostering a 
climate in the organization that makes it 
the norm to systematically track a 
program’s performance in order to learn 
what is working and what is not. 


20.100 With sustained effort, 
departments can make strides toward 
managing for results. We found in some 
cases that a focus on results has filtered 
down to the day-to-day operations, despite 
impediments such as organizational 
change. Managing for results in these 
cases has become part of doing business. 


20.101 As we reported in 1997, we found 
enough experience to draw from in 
Canada and elsewhere to establish a 


Managing Departments for Results 
and Managing Horizontal Issues for Results 


framework for managing for results. There 
is a particular need for departments in the 
federal government to: 


+ demonstrate senior management 
commitment and leadership; 


* put in place levers to encourage a 


focus on results; We found some 


progress in managing 
for results but a lot 
remains to be done 
before managing for 
results is fully 
established in 
government. 


* agree on a planning, reporting and 
accountability framework, key results 
commitments and planned results; 


* develop internal tools and capacity 
for measuring results; 


* use information on results for 
decision making at all management levels; 
and 


* report performance against stated 
expectations in a balanced manner. 


20.102 The concept of managing for 
results is now accepted across 
government. However, there is little 
practical guidance on managing for 
results, and the sharing of experience is 
limited. Departments need more 
encouragement to move beyond merely 
planning to manage for results. 
Champions need to be identified. We 
found that the leadership and support 
provided by the Treasury Board 
Secretariat has been too dispersed to be of 
enough help to departments in managing 
for results. 


20.103 Evaluation is not playing a large 
enough role in managing for results. 
Often, the only way to understand how 
programs are contributing to outcomes is 
through a systematic evaluation study. The 
federal evaluation function has been 
allowed to atrophy and to focus too often 
on operational issues rather than 
measuring how well departments are 
meeting their key results commitments. 
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Issues that cut across 
departmental lines 
must be managed for 
results. 


Managing Horizontal Issues for Results: 
Finding Solutions to the Challenges 


Introduction 


20.104 In the first part of this chapter, we 
reported on the progress five departments 
have made toward managing for results, 
and the progress in 10 of their programs. 
We found growing attention to results and 
to particular elements of managing for 
results, including planning and 
performance measurement. We concluded 
that a lot remains to be done before the 
government is fully managing for results. 


20.105 This part of the chapter looks at 
what the government is doing to support 
managing for results when several 
departments are involved in “horizontal” 
issues that cross departmental boundaries. 


Growing Attention to the 
Management of Horizontal Issues 


20.106 The commitments the federal 
government makes to Canadians are often 
targeted at broad issue areas. In recent 
Speeches from the Throne, for example, 
the government undertook to improve the 
well-being of rural Canadians, better 
integrate persons with disabilities into 
society, co-ordinate with other 
jurisdictions to address children’s issues, 
and address broad environmental issues 
like climate change and ecosystems 
management. These are horizontal issues 
— they cut across departmental lines, and 
they reflect the government’s broad vision 
for Canada. 


20.107 When a policy issue spans 
departmental mandates, no one 
department has all the levers, resources 
and expertise to manage it adequately. 
Departments must work together toward 
an overall objective and adopt a common 
vision for success. They must effectively 
manage the resources dedicated to the 


horizontal issue and consider collectively 
how to fill gaps in service and eliminate 
duplication. They must be able to 
demonstrate accomplishments and learn 
from their present performance in order to 
make progress toward the expected 
results. And the issue may require them to 
work together with other levels of 
government, and with partners outside 
government. When they do all of this, 
they are managing the horizontal issue for 
results. 


20.108 Alberta reports some horizontal 
results. Starting with the 1998 business 
planning cycle, the Government of 
Alberta Business Plan includes a section 
dedicated to key cross-government 
initiatives planned or under way in the 
three-year period covered by the Plan. In 
this way, Alberta recognizes that many of 
the pressures and challenges it faces need 
to be addressed corporately — they extend 
beyond the boundaries of single 
ministries. For the past two years, the 
Alberta Cabinet has singled out four 
cross-government or horizontal initiatives: 
two involving societal issues, and two in 
public administration. It has identified 
ministries to champion the initiatives, and 
given them the responsibility to manage 
and report on it. 


20.109 Deputy Ministers’ Task Force. 
The 1996 federal Deputy Ministers’ Task 
Force looked at the need for 
interdepartmental collaboration in the 
management of horizontal issues, and the 
challenges this represents. Its report, 
Managing Horizontal Policy Issues, made 
practical recommendations for improving 
policy co-ordination within the federal 
government. It also addressed some of the 
procedural, structural and cultural 
implications of managing horizontal 
issues. 
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20.110 In particular, the Task Force 
pointed to the need for: 


* an accountability framework to 
clarify the respective mandates and 
contributions of lead and partner 
departments; 


* consultation with stakeholders on 
cross-cutting issues; 


* realistic expectations for timing and 
outcomes, given the complexities of 
managing cross-cutting issues; 


+ adequate resources to allow the lead 
department to plan, co-ordinate and 
implement a horizontal initiative; and 


* appropriate incentives and rewards 
for collaboration and teamwork. 


20.111 In addition, the Task Force report 
discussed the importance of evaluation as 
a means of learning from experience in 
managing horizontal issues, and the 
importance of reporting and other forms 
of communication to share results and 
success factors across government. 


Horizontal Issues Addressed in 
Our Previous Audits 


20.112 The Office of the Auditor 
General and the Commissioner of the 
Environment and Sustainable 
Development have carried out a 
significant amount of work on the 
management of horizontal issues. Recent 
chapters that looked at various partnership 
arrangements recommended stronger 
frameworks for accountability and 
management, and more effort to monitor, 
evaluate and report results. (See Auditor 
General’s 1999 Report, chapters 5, 6 and 
23; and 2000 Report of the Commissioner 
of the Environment and Sustainable 
Development, chapters 2, 5, 6, 7 and 8.) 


20.113 Particularly germane to this 
discussion is the Commissioner’s 2000 
Report, Chapter 6, Working Together in 
the Federal Government. In six case 
studies, the audit identified key factors 
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affecting the success of sustainable 
development initiatives that span 
departmental boundaries. The 
Commissioner’s key findings and success 
factors are shown in Exhibit 20.8. They 
are consistent with the literature on 
managing horizontal issues, with the work 
of the Deputy Ministers’ Task Force, and 
with the findings of this study. 


Focus of the study 


20.114 The objective of our study was to 
identify the advantages of and 
impediments to using performance 
information and reporting in the 
management of horizontal issues. We 
selected case studies to explore how and 
to what extent horizontal issues are 
managed to achieve common results, 
based on the potential they offered for 
lessons learned. Most are long-standing 
horizontal issues, are clear government 
priorities, and have clear objectives. 


20.115 The following are the cases we 
selected for study: 


¢ Family Violence Initiative; 


* Government of Canada Disability 
Agenda; and 


* Canadian Rural Partnership. 


= —————e — 


The Commissioner of the Environment and Exhibit 20.8 
Sustainable Development found the following 
key problem areas: Interdepartmental 


Co-operation — Key Findings 
and Success Factors 


* unclear objectives; 

* poorly defined responsibilities; 

* unclear accountability; 

¢ weak dispute resolution mechanisms, or 
none; and 

¢ lack of senior management involvement. 


The following critical success factors were 

identified: 

* managing the impacts of participant 
turnover; 

* ensuring that departments have proper 
incentives to collaborate; 

* paying sufficient attention to the results of 
monitoring and evaluation exercises; and 


Source: 2000 Report of the 
Commissioner of the 
Environment and Sustainable 
Development, Chapter 6: 
Working Together in the Federal 
Government 


* impartial secretariat and strong involvement 
of a central agency. 
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A framework for 
managing horizontal 
issues focusses on 
co-ordination 
structures and on 
measuring and 
reporting results. 


Appendix E provides background 
information on these cases. 


20.116 We limited our study to 
collaboration among federal departments 
on clear government initiatives. We did 
not address the particular challenges of 
managing issues in collaboration with 
other jurisdictions. Other types of 
collaborative or delegated arrangements 
have already been the focus of audits in 
recent reports (in particular, the 1999 
Report, chapters 5, 6 and 23). While 
several of the observations and findings in 
those chapters can be applied by federal 
departments in managing horizontal issues 
for results, here we will discuss in 
particular the challenge and complexity of 
co-ordination. 


Observations and 
Recommendations 


20.117 Based on our review of the 
relevant literature and our case study 
work, we developed a framework for 
managing horizontal issues (Appendix F). 
The framework focusses on co-ordination 
structures suited to achieving the 
government’s objectives for an issue, and 
on elements critical to measuring and 
reporting results. The framework has five 
elements: 


* identifying an effective co-ordination 
structure; 


* agreeing on common objectives, 
results and strategies; 


* measuring results to track 
performance; 


* using information to improve 
performance; and 


* effectively reporting performance. 


20.118 As the reader will observe, this 
framework is based on the managing for 
results framework presented in 

Exhibit 20.4 in the first part of this 
chapter. 
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Effective Co-ordinating Structures 
Are Needed 


Many approaches to co-ordination 


20.119 Horizontal issues can be 
co-ordinated in a number of different 
ways; how, and how closely, will depend 
on what the issue requires. For example, 
co-ordination can be informal, limited to 
simple communication and networking 
between managers in different 
departments. Efforts at co-ordination can 
also be more formal, if departments 
recognize a need to link their common 
objectives and work co-operatively. 


20.120 Some circumstances do not 
warrant strong co-ordination. Many 
policies and programs require only 
communication among departments to 
ensure that their efforts are 
complementary. Close and structured 
co-ordination in those cases can 
overburden the participating departments 
and be counterproductive to the initiative. 


20.121 Our three cases are examples of 
structured co-ordination, each to a 
different degree. In structured 
co-ordination, departments work more in 
concert toward shared objectives, have 
shared roles and responsibilities, agree on 
a decision-making process, and co-operate 
under a defined government strategy or in 
view of a long-term public policy 
objective. 


20.122 Closer and more structured 
co-ordination is an advantage when, for 
example: 


« the issue is a high priority of the 
government; 


* the issue clearly spans the mandates 
of several departments; 


¢ there is a potential for real gains by 
integrating policy development; 


* strong champions or senior officials 
endorse the establishment of co-ordinating 
structures; 
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« the financial commitment is 
significant; or 


+ there is strong commitment and clear 
agreement among participants. 


20.123 Identifying a lead department. 
If a department is designated to lead the 
management of a horizontal issue, it has 
the critical role of ensuring that the issue 
is managed in a way that will meet the 
partners’ objectives and obligations; it 
ensures that partners are kept informed, 
that performance is monitored, and that 
partners live up to their commitments. It is 
important that the lead department be 
recognized as such by all participants and 
assigned the powers it needs to discharge 
its responsibilities. 


20.124 Each of the horizontal initiatives 
we selected has a lead department. The 
rural issue was considered significant 
enough to warrant the creation in 1996 of 
the Rural Secretariat in Agriculture and 
Agri-Food Canada. As the lead, the 
Department has a mandate to ensure that 
departments and agencies work closely 
and effectively with politicians, 
government officials and rural citizens to 
address rural problems. 


20.125 Dedicated units in Health Canada 
and Human Resources Development 
Canada have also been identified to 
co-ordinate, respectively, the Family 
Violence Initiative and the Federal 
Disability Agenda. In addition, the 
Federal Disability Agenda is supported by 
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structures including a House of Commons 
Subcommittee on the Status of Persons 
with Disabilities, and a variety of 
management and working groups created 
to direct or implement the initiative. 


20.126 The Family Violence Initiative, 
whose annual allocation of $7 million is 
shared by seven departments, is 
co-ordinated by a special unit of Health 
Canada and directed by an 
interdepartmental working group. It 
receives input from an evaluation working 
group and has accountability and reporting 
frameworks that define relationships 
among the partners. While it is well 
structured, we were informed that it 
receives little senior-level attention in the 
lead department. 


20.127 Exhibit 20.9 shows the key 
elements of co-ordination, structure and 
accountability in each case. Appendix E 
provides background information on each 
of the three case studies, along with key 
features of their co-ordination structure. 


20.128 Co-ordination takes time and 
effort. One of the issues raised 
consistently by all lead departments in our 
case studies was the time and effort it 
takes to co-ordinate effectively. For 
example, they must build consensus, 
persuade partner departments to adopt 
certain positions, ensure that decisions are 
carried out, communicate with stakeholder 
groups, and ensure that timelines are 
respected. Yet lead departments often 
have very little authority to back up their 


Family Violence Federal Disability Canadian Rural 
__Key Elements Initiative Agenda Partnership 
Co-ordination, management Co-ordinated and Co-ordinated and Co-ordinated and 
structures integrated integrated integrated 
Senior leadership Minimal Strong Strong 
ee . | a 
Accountability frameworks Formal Informal None 
J = ss" = es _ 
Reporting frameworks Formal Being Being 
developed developed 
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The lead department 
has a critical role of 
ensuring that 
objectives and 
obligations are met. 


Significant time and 
effort are needed to 
co-ordinate effectively. 


Exhibit 20.9 


Key Elements: 
Co-ordination, Structure and 
Accountability 
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Where there is senior 
leadership, horizontal 
issues are more 
focussed, better 
resourced and more 
effective at meeting 
commitments. 


Partners indicated that 
having dedicated 
resources for 
management and 
co-ordination is 
helpful. 


responsibilities. While formal agreements 
endorsed by senior officials in partner 
departments are valuable, they are not 
enough. 


20.129 Lead departments particularly 
singled out the time and effort needed for 
reporting. The lead department must, for 
example, ensure that data from many 
different sources are collected and 
collated; obtain partners’ agreement on 
reporting requirements; and engage 
partners in ongoing discussions about the 
focus, content, format, and readership of 
the report. 


20.130 Officials involved in the 
Canadian Rural Partnership and the 
Family Violence Initiative acknowledge 
that their first reporting experience was a 
challenge. In both cases, it took much 
longer than expected to negotiate the 
content of what was simply a report of 
activities, not results. Experience with 
these initial reports suggests that later, 
reporting on results will be an even 
greater challenge for management. 


20.131 Our review of the literature and 
our discussions with departments 
participating in horizontal initiatives 
indicate that having dedicated resources is 
a big help in managing and co-ordinating. 
The Canadian Rural Partnership and the 
Family Violence Initiative benefited from 
Treasury Board funding earmarked for 
co-ordination. By comparison, the recent 
federal Budget did not include funds to 
co-ordinate the Federal Disability Agenda, 
requiring the Department to reallocate 
funds internally in 2000-2001 to allow the 
Office for Disability Issues to carry out 
this responsibility. Without resources, it is 
difficult for a lead department to meet the 
full slate of commitments identified at the 
start of an initiative. However, dedicating 
resources to co-ordination does not 
guarantee success; indeed, it needs to be 
complemented by senior management 
commitment and clear engagement of all 
partner departments. 
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Tools for Issue Management 


Senior leadership is key 


20.132 Our case studies clearly 
demonstrate that support from senior 
levels of departments and clear political 
support are decisive factors in the 
successful management of horizontal 
issues. Senior officials who have a role in 
championing an issue work to ensure that 
all necessary support is available, 
including financial support. Horizontal 
issues that benefit from senior leadership 
are more focussed, better resourced, and 
more effective at meeting the 
government’s commitments. The absence 
of such support from leaders leaves 
participants drained of energy and losing 
interest in the issue. 


20.133 Leadership is needed in both the 
lead department and other participating 
departments. It is particularly important in 
cases that have no signed agreement 
among partners or a formal framework 
that clearly outlines the responsibilities of 
all parties. Even where such formal 
documents do exist, champions can do 
much to ensure that participants contribute 
to the initiative as expected and are held 
accountable for results. 


20.134 There are many ways to show 
senior leadership. For example, senior 
departmental officials who champion the 
Canadian Rural Partnership ensured that 
resources were available for co-ordination, 
engaged senior officials of partner 
departments in contributing information 
for the first annual report, and maintained 
open lines of communication with the 
co-ordination unit so that they could be 
kept informed of progress in the initiative. 


20.135 Leadership in the Federal 
Disability Agenda can be seen in the 
establishment of the House of Commons 
Subcommittee on the Status of Persons 
with Disabilities. A June 1999 report of 
the Subcommittee, Reflecting 
Interdependence: Disability, Parliament, 
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Government and the Community, 
recommended that the government take 
real action to support persons with 
disabilities. It suggested, for example, that 
the government integrate a disability 
“lens” into all policy and program 
decisions, and that it report 
comprehensively on outcomes, results 
commitments and performance of federal 
activities in the area of disability. The 
Subcommittee’s continued attention can 
Keep the issue on the government’s agenda 
for action. 


20.136 Challenges in the absence of 
leadership. For some horizontal issues, 
the general absence of senior management 
leadership is difficult to overcome. Now 
in its third phase, the Family Violence 
Initiative has seen its funding from 
Treasury Board cut back significantly, 
with a corresponding decline of senior 
management attention to the initiative in 
many of the partner departments. The lead 
department has redirected funds for 
co-ordination obtained from the Treasury 
Board to other areas in the department, 
and has not provided the support needed 
for the initiative to keep up with its 
reporting schedule. The long history of 
ground-level co-operation and dedication 
of individual officers in the lead and the 
partner departments cannot make up for 
the absence of senior management 
leadership. 


Formal frameworks are useful 
documents 


20.137 Frameworks needed to set out 
common objectives. The more ambitious 
the objectives of the horizontal initiative, 
the greater is the need for clarity of 
partners’ roles and responsibilities in 
managing, co-ordinating and reporting. 
Whether in a formal accountability 
framework document or a letter of 
agreement, all participants need to set out 
their understanding of the common 
objectives and state, in some detail, how 
and what they will contribute toward those 
objectives. 
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20.138 Such documents are particularly 
important when different partners give an 
issue different priority. In some cases, this 
occurs because some partner departments 
get Treasury Board funding for their part 
in horizontal programs and activities 
while others fund their part from their 
departmental budgets. The non-funded 
departments, for example, may resist 
having their internal activities subject to 
horizontal management. Moreover, 
non-funded departments do not have the 
formal obligations for reporting that are 
set out in Treasury Board funding 
agreements, and in a horizontal 
arrangement their reporting obligations 
are not clear. 


When the objectives 
are ambitious, it is 
important to clarify the 
partners’ roles and 
20.139 Departments in our case studies responsibilities 
noted the importance of having senior p i 
officials sign agreements so they are 
aware of the commitments their 
departments have made. While 
agreements among partners do not 
guarantee success, they are useful in 
clarifying accountabilities and 
expectations. 


20.140 Among our case studies, only the 
Family Violence Initiative had developed 
formal accountability and reporting 
frameworks. During our study, partners in 
the Federal Disability Agenda and the 
Canadian Rural Partnership were 
developing detailed reporting frameworks 
but neither had a formal accountability 
framework. When an initiative requires 
more of departments than simple reporting 
of activities, or requires that partners alter 
programs to avoid duplication or to fill 
gaps, coming to an agreement on the roles 
and responsibilities of each will be 
important. 


Finding effective levers and incentives 


20.141 Levers and incentives are needed 
to support co-operation among partners 
and to overcome pressures on each to 
focus solely on its own vertical reporting 
and accounting requirements. A wider 
range of levers is available both to lead 
and to partner departments when there are 


Levers and incentives 
are needed to support 
co-operation among 
partners. 
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formal structures and agreements. accountability structures in departments 
Departments can use communication, and horizontal commitments and 
transparency, performance appraisals, co-ordination. Current reporting tools and 
rewarding teamwork and support for processes — like reports on plans and 
horizontal issues through, for example, priorities, departmental performance 
promotion or recognition. By working in reports, and internal reporting documents 
tandem with central agencies, departments — such as business or strategic plans — are 
can make more effective use of the designed to collect information on the 
visibility given to a horizontal issue, the performance of departmental programs 
resources allocated and various human and services. They are used to support the 
resource management tools. department’s ability to demonstrate its 


accomplishments. If one partner 
department in an initiative produces 
information that is incompatible with what 
other partners produce, or if it would 
require significant resources to collect 
performance information as a special 
activity, then gathering the data and 
producing a report on collective results 
will likely be problematic. 


20.142 However, departmental officials 
we interviewed for our case studies noted 
the apparent lack of levers and incentives 
they could use to promote good 
management of horizontal issues. Central 
agencies have a role to play in developing 
and promoting such tools, if managing 
horizontal issues is a government priority. 


Challenges in Reporting 20.146 Another difficulty is getting 
partners to agree on appropriate and 
measurable indicators of performance. In 
some cases, it requires that partner 
departments overcome their resistance to 
detailed management frameworks and 
outline specific indicators, measures, and 
responsibilities for data collection and the 
collation of performance data. Of our 
cases, only the Family Violence Initiative 


20.143 Limited evidence of reporting 
or use of performance information. 
Canadians and parliamentarians want 
information on the overall 
accomplishments of these horizontal 
initiatives, collective information that 
reports on more than the individual 

Reports need to focus activities of the departments involved in 
an initiative. 


on overall has developed a reporting framework that 
; 20.144 Our cases have not yet reported outlines roles and responsibilities in the 
accomplishments of collective results. At the time of our study, Collection of data for measures that all 
horizontal initiatives two had produced reports, but both were _- Partners have agreed to. And yet it has not 
5 ty Gay reports on the activities of individual measured performance as planned and, at 
rather than individual partner departments. Departments the time this study was conducted, it had 
activities of involved in horizontal initiatives note that | Submitted only one annual report to the 
developing and using common Treasury Board Secretariat, for 1997-98. 


departments involved. performance measures is a challenging 


task, and reporting on collective 
performance cannot be expected in the 
first years of an initiative. Reporting on 
results requires adequate planning, and 
departments involved in our case 
initiatives are working now to produce 
reports on results in the future. 


20.147 A need to plan for reporting. In 
collective reporting, it is important to 
ensure that plans and expectations for 
reporting, and specifically the roles and 
responsibilities this entails, are 
commensurate with the powers and 
resources of the partners and the level of 
funding for the initiative. Reporting plans 
20.145 Conflicting priorities. We found need to consider that some departments 


that to a large extent, the challenge in must collect information from parties 
reporting on collective outcomes lies in outside the federal government: many 
the tensions between vertical horizontal initiatives involve societal 
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issues that fall under other jurisdictions or 
require the participation of parties outside 
the public sector. 


20.148 Costs not known. In Chapter 19 
of this Report, we noted that information 
on the cost of achieving results was 
generally not available in departments. In 
horizontal issues, the challenge of 
obtaining cost information is exacerbated 
by the number of departments involved 
and the different funding arrangements in 
place. A particular difficulty is knowing 
what part of the budget of non-funded 
programs contributes to the results 
expected in the horizontal initiative. For 
example, a prenatal nutrition program 
meets the needs of Canadians generally 
and, at the same time, those of segments 
of the population (such as rural Canadians 
and children). The costs of programs 
delivered to rural Canadians, particularly 
the costs that do not specifically target 
them, are hard to determine. Knowing 
(and managing) the costs of achieving 
horizontal results is difficult, but not 
impossible. Only in the Family Violence 
Initiative was there any information on the 
program dollars involved in the horizontal 
issue. 
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Lessons Must Be Learned and 
Shared 


20.149 Insufficient use of lessons. Our 
cases suggest that more effort could be 
made to use lessons from previous 
generations of an initiative. While 
partners may refer to results of past 
evaluations of the initiative, this 
information is not used routinely. In one 
initiative we examined, the turnover rate 
and the state of the collective memory 
were such that a new participant was 
unaware for some time that the initiative 
had existed for several years. 


20.150 Lead departments in the cases we 
examined are taking some positive steps 
to learn from each other’s experience. For 
example, the Office for Disability Issues 
recently invited the lead department for 
the Canadian Rural Partnership to share its 
lessons learned in reporting on a 
horizontal issue. The need to learn from 
past experience and the experience of 
others is also clear in other horizontal 
initiatives, including federal government 
commitments to green its operations, 
elements of which were recently audited 
by the Commissioner of the Environment 
and Sustainable Development. That case 
is described in Exhibit 20.10. 


| Chapter 2 of the 2000 Report of the Commissioner of the Environment and Sustainable 
Development assessed departmental progress in implementing environmental performance 
measurement for government operations. The Commissioner reported that at the time the audit 
was conducted, there was fragmented leadership with no organization given the responsibility, or 
seeing itself as the lead, for establishing a consistent consolidated and coherent federal approach 
to reporting on the greening of government operations. 


leadership role. 


In its response, the Privy Council Office agreed with the recommendation that an action plan and 
an accountability framework be developed as a way to achieve more uniform reporting of 
progress. It indicated that steps would be taken to ensure that appropriate departments play a 


In early 2000, assistant deputy ministers in three departments agreed to act as co-champions to 
co-ordinate sustainable development in government operations, namely Natural Resources 
Canada, Environment Canada and Public Works and Government Services Canada. Since then, 
the co-champions have developed a strategy to facilitate a co-ordinated approach to the 
sustainability of federal operations. They are working at the interdepartmental level to 
implement the use of common performance measures and to identify opportunities and 
impediments to achieving progress in the greening of collective operations. They have 
recognized that developing a federal approach is a challenge that requires balancing competing 
priorities, measuring performance and reporting to provide a government-wide perspective, and 
maximizing interdepartmental co-ordination of efforts to achieve results. 


Exhibit 20.10 


Applying Lessons in 
Co-ordinating Greening 
Government Operations 
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Some challenges and solutions identified 


20.151 Exhibit 20.11 summarizes some 
of the main challenges and solutions we 
identified in our review of the literature 
and in our case studies, organized 
according to the broad headings in our 
framework for managing horizontal issues 
(see Appendix F). While this is by no 
means a listing of all the challenges 
encountered by lead and partner 
departments, it does suggest what 


managing horizontal issues for results can 
entail. 


A Stronger Role for Treasury 
Board Secretariat 


The role of the Treasury Board 
Secretariat needs to be better defined 


20.152 Horizontal issues are an 
important area that requires concerted, 
strategic attention by the Treasury Board 
Secretariat. The Secretariat has been 
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Managing Horizontal Issues for Results: Challenges and Suggested Solutions 


1. Identifying an effective co-ordination structure 


¢ Partners may start out with differing commitments and priorities. Commitments, and shared understanding of roles and 
responsibilities, can be set out in an accountability framework. 


© Co-ordination takes time. Resource requirements, time and effort needed for co-ordination can be built into initiatives from the 
start to reflect the challenges or complexities involved. 


° Using effective levers and incentives. Levers can include visible support, funding, administrative instruments and human 
resource tools (such as recognition and mobility). 


2. Agreeing on common objectives, results and strategies 


¢ Shared objectives but departmental accountability. A balance is needed in the context of decentralized accountabilities through 
a detailed management framework. 


¢ Territorial protection versus co-operation. A champion can motivate participating departments to focus on horizontal results 
rather than their own separate interests. 


° Balancing objectives and resources. Early discussions between the partners and with the Treasury Board Secretariat can ensure 
that objectives are commensurate with resources available or provided, and with the political reality. 


3. Measuring results to track performance 


¢ Challenges when partnering funded with non-funded departments. It is helpful if commitments by all partners to measure and 
report on performance are clarified up front in a framework agreement, and supported by senior-level leadership. 


° Vertical and horizontal information needs. Early discussions between the partners and within participating departments can 
ensure that both vertical and horizontal reporting requirements are addressed. 


Attribution preoccupies participants in horizontal initiatives. Evaluation subcommittees have a role to play in establishing 
results-based management and accountability frameworks, explaining how the contributions of various partners can be accounted 
for and developing approaches to measurement. 


Reporting requirements need to be reasonable. Partner departments and the Treasury Board Secretariat can work together to 
ensure that reporting requirements are balanced with resources and objectives. 


4, Using information to improve performance 


Data from too many sources are hard to collate. It is sometimes best to start by measuring a small number of indicators 
focussed on high-level results, and build from there. 


Limited use of lessons, evaluations and monitoring. Responsibilities for using performance information in planning and 
implementation must be clearly outlined. 


5. Effectively reporting performance 


Negotiating the reporting framework takes time. Agreement on reporting poses a challenge in a time-limited initiative. Central 
agencies can ensure that timeframes are considered and can help partners reach consensus. 
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proactive in identifying programs that 
could be managed horizontally. It has 
made presentations and developed draft 
guidelines on reporting and accountability 
frameworks for managing horizontal 
issues, and on its Web site it has shared 
information on collective and societal 
results and indicators. The Secretariat has 
also supported individual initiatives. Lead 
departments in horizontal initiatives have 
indicated that they value its contribution. 


20.153 The Treasury Board Secretariat 
requires that submissions for funding of 
horizontal initiatives contain a 
commitment to performance reporting and 
evaluation. However, not all horizontal 
initiatives receive funding through the 
submission process. And, as already 
noted, not all departments receive funding 
for their part of the horizontal initiative. 
For those that do not, there are no central 
requirements to report collectively or to 
participate in collective evaluations. It is 
important for the Secretariat to ensure that 
horizontal issues are managed and 
co-ordinated adequately and resourced in 
a way that facilitates evaluation and full 
reporting when funded and non-funded 
departments are partners in a horizontal 
initiative. 


20.154 We have observed that the role 
played by the Treasury Board Secretariat 
in the management of horizontal 
(interdepartmental) issues is piecemeal 
and differs from one initiative to the next, 
apparently independent of the 
characteristics of the initiative. The 
Treasury Board Secretariat does not have 
a strategic approach for dealing with the 
fact that a growing number of issues are 
addressed horizontally. 


20.155 In view of the many challenges to 
horizontal initiatives that we have 
discussed in this chapter, we support the 
recommendations in the Deputy Ministers’ 
Task Force report, Managing Horizontal 
Policy Issues, and in several reports of the 
Commissioner for the Environment and 


Managing Departments for Results 
and Managing Horizontal Issues for Results 


Sustainable Development. These include 
recommendations that central agencies: 


Horizontal issues 
require concerted, 
strategic attention by 
the Treasury Board 
Secretariat. 


* clarify expected outcomes and 
accountabilities of the partners involved, 
and 


* provide advice on the co-ordination 
and management of horizontal issues. 


20.156 We also agree with the 
Commissioner’s recommendation that the 
Treasury Board Secretariat prepare a “best 
practices” guide for interdepartmental 
co-ordination, with input from those who 
have worked on horizontal issues. 


20.157 Stronger central leadership is 
needed if the government wants to 
continue to promote horizontal issues as a 
priority and to focus increasingly on 
horizontal policy development, 
implementation and accountability. 


20.158 The Treasury Board Secretariat 
should play a stronger leadership role to 
ensure that where appropriate, 
horizontal issues are managed for 
results. This includes ensuring that 
resources are available for the 
co-ordination and management of 
horizontal issues that are government 
priorities, and ensuring that external 
reporting takes place. 


20.159 The Treasury Board Secretariat 
should play a stronger role in 
communicating good practices and 
lessons learned in managing horizontal 
issues for results. 


Conclusion Government is still at 
the start of its move 
toward reporting and 
using performance 
information in 


managing horizontal 


20.160 Our case studies indicate that 
there are many challenges in managing 
horizontal issues for results. Partners in 
horizontal initiatives can co-ordinate 
sufficiently to identify collective 
objectives, to share information, to 
develop frameworks for accountability 
and to report on activities. However, it is 
more difficult for them to manage tasks 
that require them to compromise or 
realign their own programs. Managing for 


issues for results. 
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results is difficult without sufficient 
resources; funds and human resources are 
needed to co-ordinate the efforts of 
partners and to implement reporting 
frameworks that can show Canadians and 
parliamentarians what has been 
accomplished by a horizontal initiative. 


20.161 Government is still at the start of 
its move toward reporting and using 
performance information in managing 
horizontal issues for results. The Treasury 
Board Secretariat in particular has a role 
to play in supporting the effective use of 
levers and incentives by departments 
involved in horizontal initiatives, and in 
creating the conditions that favour more 


effective management of horizontal issues. 


Government’s response: While agreeing 
with the emphasis the government has 
placed on managing for results, the 
Auditor General concludes that the 
fundamental shift in perspective from 
process-oriented or results-oriented 
management has not yet occurred 
generally across the public service. The 
chapter highlights the need to strengthen 
measurement of results, in particular the 
evaluation function. In addition, the audit 
notes the particular challenges in 
managing horizontal issues and 
recommends a stronger role for the 


Treasury Board Secretariat in supporting 
departments in this regard. 


With the tabling of Results for Canadians: 
A Management Framework for the 
Government of Canada in Parliament in 
March 2000, the government now has a 
framework to move forward in this area. 
As the government implements this 
framework, departments and agencies will 
need to improve their measurement of 
results and costs and use this information 
to deliver better results for Canadians. To 
ensure that these expected improvements 
take place, the Treasury Board Secretariat 
continues to monitor implementation and 
upgrade its plans to exercise appropriate 
leadership and support to departments 
and agencies in this endeavour. It also has 
made internal managerial changes to 
promote the effective implementation of 
this management framework. 


The government recognizes that managing 
horizontal issues, including those 
involving other jurisdictions, is a 
challenge for departments. The Treasury 
Board Secretariat will continue to actively 
track major collaborative initiatives and 
ensure that lessons-learned are shared. It 
is also examining how to exercise the best 
possible leadership and support in 
responding to these new challenges. 
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% About the Audit 


Objectives 


The objectives were to: 


* examine the extent to which selected departments were managing for results, including their use of 
information on performance to manage programs; 


¢ identify lessons learned in implementing managing for results; 


¢ assess the leadership the Treasury Board Secretariat has provided to support managing for results 
throughout government; and 


¢ in selected cases, to identify the advantages of and impediments to the use of performance information 
and reporting in the management of horizontal issues. 


Scope 


We examined five large departments — Agriculture and Agri-Food Canada, Correctional Service Canada, 
Environment Canada, Industry Canada, and Natural Resources Canada. In each department, we asked senior 
officials and/or corporate staff to help identify areas where the use of performance information could be 
expected. We selected and then examined a total of 10 programs/business lines to determine to what extent 
performance information is actually used and to what extent other elements of managing for results are in 
place. We also asked the departments to identify the lessons they learned in implementing their regimes for 
managing for results. 


In addition, we followed up on the findings and recommendations of our 1996 Report, Chapter 3, Federal 
Program Evaluation. We examined the same departments selected for examination in our current audit. We 
looked at how these departments are planning for evaluations and using the information they provide on 
performance to manage for results. 


For our study discussed in the second part of this chapter, we used three cases to explore good practices in 
managing horizontal issues for results and reporting results. Our selection of the case studies was based on the 
potential they offered for lessons learned, given that they are long-standing horizontal initiatives, are clear 
government priorities, and have clear objectives. 


Criteria 


To assess use made in selected departments of information on performance to manage programs and reporting 
on them: 


° In selected departments, we expected to see at the corporate (or department-wide) level appropriate use 
of information on performance to improve program operations and design, to update planned results and 
budgets, and to report program performance at the corporate level. 
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° Jn selected programs/business lines, we expected to see appropriate use of information on performance to 
improve program operations and design, and to update planned results. At a minimum, we expected to 
see elements of managing for results in the programs and business lines we examined. These elements 
include fostering an organizational climate that encourages managing for results; measuring performance; 
and planning a strategic framework that demonstrates agreement on areas of key results, performance 
measures, and performance expectations. 


In assessing the leadership provided by the Treasury Board Secretariat to support managing for results 
throughout government, we expected to find that the Secretariat: 


* provides appropriate and consistent direction, guidance and feedback to departments on their efforts at 
managing for results; 


¢ gathers and disseminates best practices and lessons learned; 
* uses departmental results and performance information; and 


* encourages departments and agencies to institute managing for results in a reasonable time frame. 


In examining the management of horizontal issues for results, we reviewed the applicability of the framework 
we had developed for managing for results and looked for lessons identified in our cases. As this component 
of our work was a study, we did not set out specific criteria against which the cases were assessed. 


Approach 


We interviewed senior managers in the five selected departments and in the Treasury Board Secretariat 
(TBS), visited several field operations, and gathered relevant documents. We reviewed the literature in other 
jurisdictions on managing for results. 


In our examination of the leadership role played by the Treasury Board Secretariat, we interviewed TBS 
officials, examined relevant documents and interviewed departmental officials for their perceptions of the 
Secretariat’s leadership. 


In our study of managing for results in selected horizontal initiatives, we interviewed departmental 
participants, including officials in lead departments. We had discussions with officials of the Treasury Board 
Secretariat, collected relevant documentation, and reviewed literature on horizontal issue management. 


Audit Team 


Assistant Auditor General: Maria Barrados 
Principal: John Mayne 
Director: Patricia MacDonald 


Christina Brooks 
Allison Fader 
Adéle Lamoureux 
Erin Molloy 

Tom Wileman 


For information, please contact John Mayne. 
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Appendix B 


Follow-up on Our 1997 Cases 


Cases 


Managing Departments for Results 


and Managing Horizontal Issues for Results 


Key Findings in 1997 


Follow-up Findings 


_The North American Waterfowl 
_Management Plan (Environment Canada) 


_An international (U.S./Canada/Mexico) 

waterfowl and wetlands conservation program, 
implemented through joint ventures involving 

| national and provincial/state governments, 
non-governmental organizations and the private 
sector. 


* Results measured to improve 


performance. 


Organizational strengths 
focussed on use of internal 
and external levers. 


* Clear and measurable targets are established 
for waterfowl population and habitat. 


® Performance expectations for habitat 
conserved and results achieved are reported 
annually. 


°® Management structure promotes 


communication and ensures commitment 
from all partners. 


* Updates, informed by evaluation studies, 
continue to keep the Plan responsive and 
relevant. 


| Investigation and Control (Human Resources 
Development Canada) 


Prevents, detects, and deters fraud and abuse of 
| . 
| the employment insurance program. 


+ 


Supporting incentives for 
organizational climate 
established. 


Key performance indicator 
used for performance 
measurement and program 
adjustment. 


® Performance agreements include 
performance expectations and are used in the 
assessment of managerial performance. 


* Consistent measurement and reporting 


against the key performance indicator — total 
savings — identified in 1997. 


*® Resources allocated based on the results 
achieved. 


Canada Centre for Mineral and Energy 
Technology (Natural Resources Canada) 


Conducts and sponsors research for the 
economical, safe, and environmentally 
responsible recovery and use of mineral, metal, 
and energy resources. 


Organizational focus on 
outcomes. 


Supporting incentives for 
organizational climate 
established. 


* Significant improvements in business 
planning; three-year business plans 
incorporate a results chain. 


© Leadership to manage for results pushed 
down to program management level. 


® Performance agreements set performance 


expectations and are directly linked to 
business plans. 


‘Travellers Program (Canada Customs and 
Revenue Agency) 


Processes and controls the entry of persons and 
their baggage into Canada through legal ports of 
entry. 


Key indicators used to 
improve program results. 


Supporting incentives for 
organizational climate 
established. 


* Stints — a performance measurement 
(sampling) system — are still being used to 
measure program performance. 


* Performance information is used to make 
program adjustments such as reallocation of 
staff among ports. | 


Performance agreements for senior managers | 
specify program delivery targets. 
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Initiatives Supporting Managing for Results 


Government Initiative Implications for Managing for Results 
foes 
Modernization of Modernization of Comptrollership aims to integrate into a coherent whole, recent 


Comptrollership pilot project | government management practices and priorities for change. The initiative is intended to 
produce performance information linked with relevant financial information for sound 
resource management and effective decision making to achieve results. Performance 
information will reflect business planning, risk management, costing, budgeting, 
evaluations, accounting, and non-financial and financial performance reports. 


Treasury Board President’s A description of the President’s annual report to Parliament on Managing for Results is 
annual report to Parliament provided in Chapter 19. Treasury Board has used this document to insist on the 
on Managing for Results importance of a focus on results, and sets out actions the government plans to take to 


promote capacity in departments, managing for results in horizontal issues, and 
improved reporting. 


Improved Reporting to The government launched the first phase of its Improved Reporting to Parliament 
Parliament Project Project (IRPP) in 1995. The intent of the project was to provide Parliament with better 
information on the federal government’s plans and performance with a focus on results. 
On a pilot basis, and with the agreement of Parliament, Part III of the Main Estimates 
was split into two documents: the fall Departmental Performance Reports and the spring 
Reports on Plans and Priorities. The government is moving into the second phase of 
IRPP to enhance the involvement of parliamentarians in the reporting of performance 
information to Parliament. 


Financial Information The Financial Information Strategy (FIS) seeks to fundamentally change the way in 
Strategy which the government manages its financial information. The vision for FIS is to 
enhance the government’s decision making and improve organizational performance 
through the strategic use of financial information. The Strategy was approved by 
Treasury Board in 1995, and is expected to be fully implemented by 1 April 2001. 


Results for Canadians The document sets out a framework for government management and an agenda for 
(March 2000) change that focusses on the achievement of results. Commitments are made to manage 
for results: all departments must be focussed on the achievement of results, and on 
reporting them in simple and understandable ways. The government will continue its 
movement toward a results-based approach that can distinguish program strengths and 
weaknesses, and provide guidance on what does and does not work. 


Practical Guidelines to This document provides practical advice on what to include in Treasury Board 
Preparing Treasury Board submissions. In support of managing for results, submissions are expected to outline 
Submissions (May 2000) outcomes, state how the initiative will be monitored and evaluated, include an 


performance and accountability framework, or a commitment to developing one at a 
later date. The framework reflects a full managing for results cycle, including 
performance measurement, reporting, and evaluation. 
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Appendix D 


Status of Our 1996 Recommendations 


on Federal Evaluation 


Recommendations 


Comments 


Treasury Board Secretariat should: 


Ensure that there is a clear statement of specific government 
evaluation priorities. It should monitor departmental business plans 
for gaps and omissions with respect to government priorities, and 
should take steps to deal with gaps and omissions (paragraph 3.74). 


Unsatisfactory progress. The Secretariat has not stated the 
government’s priorities for evaluation. It sets out evaluation 
requirements in selected areas. 


Ensure that its report to Parliament credibly represents the 
| performance of review and includes specific measures on [the state 
of] evaluation. The report should include the government’s 
evaluation priorities and progress in addressing them (3.86). 


Recommendation rejected. The President’s annual report to 
Parliament does not report on the state of evaluation. It has evolved 
to a broader scope, focussing on reporting and managing for results. 


| 
Make clear the quality standards to be applied for different types of 
_ effectiveness measurement and analysis (3.106). 

| 


Unsatisfactory progress. The Secretariat has not set out quality 
standards for evaluation and review. A proposed new Treasury 
Board evaluation policy does set out standards, and clarifies 
evaluation and review. 


Deputy heads should: 


[ 


Ensure that departmental business plans include evaluation findings 
_and identify priorities for evaluation (3.64). 


Satisfactory progress*. Departmental business plans are no longer 
required by the Secretariat, but planning exercises occur for business 
lines. Planning documents often include performance measurement 
strategies and reflect evaluation plans. 


Ensure that plans for evaluation are developed as part of their 
| business planning (3.96). 


Satisfactory progress*. Plans for evaluation are organized by 
business line. Departments are working to further integrate | 
departmental priorities in their evaluation plans. | 


Key: 


Fully addressed — The original audit finding has been fully addressed and no further action is needed. Our Office will not follow up 


further. 


Satisfactory progress — Reasonable progress has been made in addressing the original finding, but some additional action is required. 


Our Office will do further follow-up work. 
Unsatisfactory progress — Reasonable progress has not been made 
required. Our Office will do further follow-up work. 


* Based on an examination of the five departments. 


Source: Report of the Auditor General of Canada, Chapter 3, Evalu 


in addressing the original finding and considerable further action is 


ation in the Federal Government, 1996 
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Horizontal Issues — Three Case Studies 


Features 


Family Violence Initiative 


Federal Disability Agenda _ | Canadian Rural Partnership | 


Lead department 


at 


Health Canada 


Human Resources Development 
Canada 


Agriculture and Agri-Food 
Canada 


Start and end dates 


1997-98 to 2001-02 


The initiative is now in its third 
phase. 


1999-2000 (no end date) 


This is the second formal 
initiative, following the National 
Strategy for the Integration of 
Persons with Disabilities. 


1998-99 to 2002-03 


A Secretary of State (Rural 
Development) was appointed in 
1999. 


Number of participants 


12 


23 


29 


Federal contribution 


$35 million over five years for 
seven departments. 


Five other departments address 
family violence issues through 
their existing programs and 
activities. 


Some elements of the Federal 
Disability Agenda receive 
support: 


$ 75 million a year 
Opportunities Fund and tax 
assistance (HRDC) 


$11.5 million over three years 
for the Health and Activity 
Limitation Survey (Statistics 
Canada). 


Other departments and agencies 
address disability issues through 
their existing programs and 
activities. 


$20 million over four years for 
the Canadian Rural Partnership, 
which includes co-ordination 
activities and a grants and 
contributions program. 


Other departments and agencies 
address rural issues through their 
existing programs and activities. 


Co-ordination 
mechanism 


Family Violence Prevention Unit 


Office for Disability Issues 


Rural Secretariat 


Agreements and 
framework documents 


Accountability framework 
Reporting framework 


(both are Treasury Board 
requirements) 


Three-Year Strategic Plan 


Accountability framework 
(informal). 


Reporting framework (being 
developed). 


Interdepartmental strategic 
action plan (being developed). 


Interdepartmental performance 
measurement framework (being 
developed). 


Structures used for 
co-ordination 


Interdepartmental Working 
Group 


Evaluation Working Group 


House of Commons 
Subcommittee on the Status of 
Persons with Disabilities. 


Assistant Deputy Ministers’ 
steering committee. 


Interdepartmental working 
group. 


Various working groups 
(evaluation, Aboriginal issues). 


Interdepartmental working 
group. 


Rural teams in all provinces and 
territories. 
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Appendix F 


Framework for Managing Horizontal Issues 


Effectively managing a horizontal issue involving a number of departments to achieve specific results requires the following: 
1. Identifying an effective co-ordination structure 
Effective co-ordination is essential to managing horizontal issues and requires: 
* Establishing roles and responsibilities. Agree on the roles and responsibilities of departments involved and who will take 
the lead. 


* Developing an appropriate level of co-ordination. Define in a framework document the management and governance 
structure, and put in place the co-ordination structure as appropriate 


* Promoting co-ordination through levers and incentives. Structures and frameworks are not enough. A range of levers and 
incentives should be available, including visible political support, funding, and human resource tools. 


2. Agreeing on common objectives, results and strategies 


Reasonable agreement is needed among the participants on the overall objectives, what more specifically will be accomplished over 
what time frame, and what and how departments will contribute. This involves: 


* Defining common objectives. Determine the objectives that will guide departmental actions and provide a basis for a 
common vision of success. 


* Agreeing on performance expectations for the initiative. Agree on specific results and a set of performance expectations 
for the horizontal issue that are realistic and challenging, taking into consideration the priority of the issue area and the 
co-ordination structure in place. 


* Defining performance indicators for the initiative. Agree on a manageable set of qualitative or quantitative performance 
indicators. These indicators should be relevant for each department as well as for collective action. 


* Agreeing on how departments will contribute. Articulate strategies and performance expectations for departments that are 
commensurate with the resources available and with political/jurisdictional reality. 


3. Measuring results to track performance 
Managing the horizontal issue for results requires: 


* Measuring performance. Practices exist for departments involved to collect, assess and report on performance and costs, 
using the indicators identified and agreed on. 


* Evaluating programs. Practices exist for evaluating the relevant components of the horizontal initiative in the different 
organizations. 


* Reviewing and updating indicators and measures. Based on experience gained, changing circumstances, and experience 
gained in other horizontal issues, make periodic changes and improvements to indicators, expectations and measurement 
approaches. 


4. Using information to improve performance 


Realizing the benefits of managing for results requires: 


* Using performance information. The performance information gathered and assessed is used to improve management of, 
and performance in, the horizontal issue, such as adjusting the co-ordination mechanism or the activities undertaken. 


* Identifying good practices. Based on what works and what does not, identify and use lessons and good practices to improve 
performance. 


5. Effectively reporting performance 
Reporting among the departments involved and to other stakeholders requires: 


* Defining a reporting regime. Define a reporting regime and a communication strategy that reflect the level of priority given 
to the horizontal issue, and its size, including the resources it has available. The regime and strategy should also meet 
departmental, horizontal and stakeholder information needs. 


* Describing the context and strategies used. State clearly in reporting documents the rationale and objectives of the 
collective effort, its major strategies, and its related external context. 
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Managing Departments for Results 
and Managing Horizontal Issues for Results 


¢ Stating meaningful performance expectations. Clear and concrete key performance expectations focussed on outcomes. 

¢ Reporting performance accomplishments against expectations. Key accomplishments are related to expectations and 
attributable to individual or collective efforts. 

¢ Demonstrating the capacity to learn and adapt. The report demonstrates the ability of the collective effort to learn from 
past performance and adapt to external changes. 

¢ Reporting on the fairness and reliability of the performance information. Report performance information that is fair and 
reliable. 
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Chapter 21 


Post-Secondary Recruitment 
Program of the Federal Public Service 


The audit work reported in this chapter was conducted in accordance with the legislative 


mandate, policies and practices of the Office of the Auditor General. These policies and 
practices embrace the standards recommended by the Canadian Institute of Chartered 


Accountants. The numbered paragraphs in bold face represent recommendations. 
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Post-Secondary Recruitment 
Program of the Federal Public Service 


Main Points 


21.1 Our review of the Post-Secondary Recruitment program in the federal public service found that the Public 
Service Commission of Canada is recruiting qualified candidates. But there are too few of them given the 
significant number of public service executives, professionals and managers who will be eligible to retire in the 
near future. The government has to immediately address the recruitment issue so that it can continue to deliver 
programs and provide services of quality to Canadians. 


21.2 We found weaknesses in the program’s efficiency and effectiveness in the following areas: 


¢ Forecasting of recruitment needs is not clearly spelled out. Without a clear idea of the number and 
type of employees needed, it is difficult to establish clear recruitment levels and develop appropriate 
recruitment strategies. 


* The recruitment levels under the program have not been met consistently. 


¢ In light of the increased competition for university graduates, the Commission needs to be a more 
aggressive recruiter of graduates. There has to be better promotion of the public service as a career 
choice, increased visibility on campus and improvement in scheduling and timeliness of recruitment. 


* Very few departments participate in the program, which limits its capacity to renew the public 
service. 


* The Commission does not establish and manage inventories of qualified candidates for departments. 


¢ The program and its activities have not been evaluated consistently and systematically to improve 
their effectiveness and efficiency. As a result, it makes it difficult for the government to take 
appropriate corrective action on recruitment. 


* Reporting of results is not adequate. 


21.3 We are concerned about the government’s ability to address the recruitment challenge. 


Background and other observations 


21.4 The public service has become significantly older. Seventy percent of executives could retire by 2008, 
and the feeder groups, the people who could eventually replace them, have a similar retirement profile. Public 
servants under 35 years of age are underrepresented. 


215 For existing senior executives, it took an average of 10 years for them to move from a professional entry 
level to an executive level. Recruitment has, therefore, become imperative to ensure a well-functioning public 
service in the future. 

21.6 The public service has changed and requires a higher proportion of knowledge workers than in the past. 


Today, these workers constitute 55 percent of public servants. 


21.7 Recruitment has been an issue in the public service for at least 10 years. Many reports, and more recently 
the 1999 Speech from the Throne, have reported the need to recruit for the public service because of 
demographics and other factors, such as increased competition for qualified candidates. 
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21.8 It is recognized that an effective and non-partisan public service is essential to the strong functioning of a 
democratic country. To remain a vibrant institution, the public service must be able to recruit and retain 
well-qualified employees. 


21.9 The government recognizes that recruitment in the public service is a priority. In his March 2000 annual 
report, the Clerk of the Privy Council stated the following: “We need to start recruiting people now so that we 
have the time to groom and nurture skills and so new recruits will have time to benefit from the depth of 
experience and knowledge of public servants across the country.” 


21.10 We are concerned about the government’s ability to address the urgent challenge of recruitment for the 
years ahead. 


The responses of the Treasury Board Secretariat, on behalf of the government and the departments audited, 
and of the Public Service Commission are included at the end of this chapter. The responses recognize the 
importance of recruitment and indicate plans to address the issues identified. 
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Introduction 


Entering the federal public service 


21.11 Staffing in the federal public 
service, which includes initial 
appointments as well as promotions, is 
administered by the Public Service 
Commission of Canada under the Public 
Service Employment Act. The Act covers 
20 departments and some 60 agencies, 
which form the core public service. 


21.12 To hire new people in the public 
service, departments and agencies must 
have vacant positions and have allocated 
funds for these positions. Initial hiring into 
the public service can be done in a variety 
of ways. The majority of external hiring is 
done through individual competitions to 
fill vacant positions, as required. This 


Post-Secondary Recruitment Program of the Federal Public Service 


practice is known as “general recruitment” 
(see Exhibit 21.1). It is not a structured 
program but a series of specific 
departmental staffing actions. 


21.13. The Post-Secondary Recruitment 
program is a structured program. It is the 
government’s main recruitment vehicle for 
a wide variety of entry-level positions in 
the Administrative and Foreign Service, 
and Scientific and Professional categories. 
(See the list of occupational groups in 
these categories in Appendix A.) 


21.14 The public service has 
traditionally hired at the entry level and 
promoted from within. Indeed, section 11 
of the Act states that “appointments shall 
be made from within the Public Service 
except where, in the opinion of the 


The Post-Secondary 
Recruitment program 
is a structured 
program. It is the 
government’s main 
recruitment vehicle for 
a wide variety of 
entry-level positions. 


Exhibit 21.1 


Recruitment in the Federal Public Service 


General recruitment 


Components of the Post-Secondary Recruitment program 


Individual competitions for . 
vacant positions in departments 
| and agencies for all categories, 


occupational groups and levels agencies 


Specific entry-level | e 
positions for 
departments and 


Entry-level positions for e 
departmental development 
programs: 


= Foreign Service Officer 
Development program for the 
Department of Foreign Affairs 
and International Trade 


= Economics Recruitment and 
Development program for 
Statistics Canada 


Entry-level positions for 

corporate development 

programs: 

= Management Trainee 
Program (MTP), managed by 
the Public Service 
Commission 


Accelerated Economist 
Trainee Program (AETP), 
managed by the Commission 


= Financial Officer Recruitment 
and Development program 
(FORD), managed by the 
Treasury Board Secretariat 


Candidates can apply during 
the posting period, which 
varies between 10 and 15 


working days. | longer. 


Positions are posted through 
the Public Service 
Commission’s Web site. 


Students can apply during the campaigns of Post-Secondary Recruitment program. The fall campaign 
usually runs for four weeks in September and October. The winter campaign usually runs for four 
weeks in January. The application period for some corporate development programs may run a little 


Source: Public Service Commission of Canada 
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Commission, it is not in the best interests 
of the Public Service to do so.” 


21.15 The positions in the Scientific 
and Professional category require a 
university degree, as do almost half of 
those in the Administrative and Foreign 
Service category. Managers and 
executives are usually selected from all 
these categories, which, including the 
Executive Group, comprise what are often 
referred to as “knowledge workers.” 


21.16 Because the nature of the public 
service’s work has shifted, these 
categories make up an increasingly larger 
proportion of the public service’s work 
force. Today, they represent about 

55 percent of the work force, up from 
about one third in the mid-1980s. 
Therefore, replenishing these categories is 
critical to a well-functioning public 
service. 


21.17 The Public Service Commission 
manages the Post-Secondary Recruitment 
program, which has three components: 
recruitment for specific jobs in 
departments and agencies, recruitment for 
departmental development programs, and 
recruitment for three corporate 
development programs (see Exhibit 21.1). 
The specific jobs in departments include 
economists and policy analysts, nurses, 
commerce officers, and information 
technology specialists. Departmental 
development programs include the 
Foreign Service Development Program at 
the Department of Foreign Affairs and 
International Trade, and the Economists 
Recruitment and Development program at 
Statistics Canada. The corporate 
development programs comprise the 
Management Trainee Program (MTP), the 
Accelerated Economist Training Program 
(AETP) and the Financial Officer 
Recruitment and Development (FORD) 
program. They are designed to move 
candidates through to middle management 
or professional levels quickly. 


An aging public service work force 


21.18 The public service has become 
significantly older over the past eight 
years. This characteristic is particularly 
more pronounced in the Executive, 
Administrative and Foreign Service, and 
Scientific and Professional categories, 
where some 45 percent of the work force 
is aged over 45. In contrast, only 

18 percent of public servants in these 
categories are under 35. Part of the reason 
for these figures is that during Program 
Review, many experienced employees as 
well as younger ones left under a 
departure incentive program, and hiring 
was extremely limited. 


21.19 The Treasury Board Secretariat 
expects the annual separation rates of the 
Executive category to increase from 

six percent in 1999-2000 to almost 

10 percent after 2002-03. In April 2000 
we reported that 70 percent of executives 
could retire without any penalty by 2008. 
To be eligible for retirement without a 
penalty, an employee must be aged 55 and 
have 30 years of pensionable service, or 
be aged 60 with a minimum of two years 
of pensionable service. 


21.20 Data show that the “feeder 
groups” (the traditional pool of candidates 
for executive positions) are also older than 
the average age of the public service. As 
well, this pool will see higher departure 
rates in the next decade, which will 
increase the difficulty of replacing many 
executives (see Exhibit 21.2). Data from 
the Treasury Board Secretariat’s 
information system indicates the 
following: 


* inthe primary feeder group, which 
accounts for half of promotions to the 
Executive category, 48 percent of 
employees will be eligible to retire 
by 2008; and 


* in the secondary feeder group, which 
accounts for almost one quarter of 
executive appointments, 53 percent of 
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employees will be eligible to retire 
by 2008. 


21.21 Our Office and other 


organizations continue to express concerns 


about the need to recruit knowledge 
workers and to improve recruitment 
practices. For instance, in 1998 we 


reported on the impact of staff shortages 


on the management of the International 
Tax Directorate. In 1999 we noted the 
need for more flexible recruitment 
practices for the science and technology 
community. This year we continue to 


examine ongoing staff shortages and their 


80— 


70— 


Executive 
group group 


The primary and secondary feeder groups comprise senior classification levels in the Scientific and 


47.9 


Primary feeder 
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overall impact on service quality and 
program delivery in financial management 
(see our October 2000 Report, Chapter 13) 
and health and safety (see our 

December 2000 Report, Chapter 24). 


Focus of the audit 


21.22 The purpose of our audit was to 
determine the extent to which the 
Post-Secondary Recruitment program 
would help the government renew its 
aging professional and management work 
force and to assess the effectiveness and 
efficiency of the program. 


Exhibit 21.2 


Retirement Eligibility of 
Executive, Primary and 
Secondary Feeder Groups 


(percentage) 


March 2001 ES 


Secondary feeder March 2005 a 


group 
March 2008 [i 


Professional, and Administrative and Foreign Service occupational categories (see table below). 


Primary feeder group 


Secondary feeder group 


Scientific and Professional 


e Economics, Sociology and Statistics (ES 06, 
ES-07) 


Scientific and Professional 
e Auditing (AU-05) 


e Engineering and Land Survey (ENENG-0S, 
ENENG-06) 


e Physical Sciences (PC-04, PC-05) 
e Scientific Research (SEREM-02) 


Administrative and Foreign Service 
e Administrative Services (AS-06, AS-07) 
e Foreign Service (FS-02) 


e Program Administration (PM-05, PM-06) 


Administrative and Foreign Service 

e Commerce (CO-03, CO-04) 

e Computer Systems Administration (CS-04, 
CS-05) 

e Financial Administration (FI-03, FI-04) 

e Information Services (IS-06) 


e Personnel Administration (PE-05, PE-06) Source: Treasury Board Incumbent 
| File (excluding the Canada Customs 


e Purchasing and Supply (PG-06) 
; tga Revenue Agency) — Not audited 
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We continue to find a 
clear lack of 
co-ordination and 
direction in dealing 
with the government’s 
recruitment priority. 


21.23 We looked at the program as the 
government’s main structured program for 
recruiting university graduates in the 
Administrative and Foreign Service and 
Scientific and Professional categories. We 
examined how the Public Service 
Commission and the Treasury Board 
Secretariat manage recruitment from a 
central perspective. We also examined 
how the Department of Foreign Affairs 
and International Trade, Fisheries and 
Oceans, and Statistics Canada identified 
their human resource requirements and 
made use of the Post-Secondary 
Recruitment program. 


21.24 We selected these departments 
based on the Public Service Commission’s 
information, which ranked them as key 
users of the program in 1998-99. 


21.25 We reviewed three main phases 
of the recruitment process: planning, the 
management of recruitment activities, and 
the assessment and reporting of results. 


21.26 During our audit we interviewed 
senior human resource managers from 
various private and para-public 
organizations to learn about their 
recruitment practices. We also spoke to 
university placement officers to discuss 
their views on the Post-Secondary 
Recruitment program. 


21.27 We present more details in About 
the Audit at the end of this chapter. 


Observations and 
Recommendations 


Lack of Corporate Direction 
Hampers Recruitment Planning 


21.28 The 1998 Committee of Senior 
Officials’ (COSO) subcommittee report on 
the human resource community called for 
a more strategic approach to human 
resources management. In 1998, La 
Reléve noted the need for a human 
resource strategy for the public service to 


retain more employees and focus 
recruitment in areas of skill shortages. 


21.29 We continue to find a clear lack 
of co-ordination and direction in dealing 
with the government’s recruitment priority 
partly because of unclear roles and 
responsibilities. 


21.30 Under the Financial 
Administration Act, the Treasury Board 
determines the size of the public service 
and ensures that human resources are 
allocated and used effectively. It aims to 
perform a strategic and results-oriented 
management role as the government’s 
management board. 


21.31 The Public Service Commission’s 
mandate is to ensure that Canada has a 
highly competent, non-partisan public 
service that is representative of the 
population it serves. Over the years, the 
Commission has delegated much of its 
staffing responsibilities to departments 
and adopted an advisory and oversight 
role. However, the Commission has 
retained authority for initial appointments 
to the public service and responds to 
requests from departments for recruitment 
from outside the public service. 


21.32 Departments are responsible for 
determining the resources they need for 
program delivery and obtaining funds and 
guidance from the Treasury Board 
Secretariat to address resource shortages. 
They must work with the Commission to 
recruit new staff. 


21.33 We expected the Secretariat and 
the Commission, which have shared 
responsibilities for human resource 
management, to take lead roles on 
recruitment. 


21.34 The Clerk of the Privy Council, 
as Head of the Public Service, established 
three priorities for human resource 
management in his Seventh Annual 
Report to the Prime Minister in 

March 2000: recruitment, workplace 
well-being, and learning and development. 
He established three subcommittees of 
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COSO to work on these priorities. In the 
spring of 2000 the Clerk asked all deputy 
ministers about their plans in these three 
critical areas for the short, medium, and 
long term, and what corporate support or 
help they needed. 


21.35 The Clerk chaired the 
subcommittee on recruitment, which 
indicates the priority he placed on this 
area. The July 2000 subcommittee report 
outlined expected behaviours for all 
managers, highlighted some best practices 
for other departments to consider, and 
attached a general plan for central 
agencies. (Appendix B contains highlights 
of the report, which deals with the broad 
issue of recruitment, not just the 
Post-Secondary Recruitment program.) 


21.36 In response to the Clerk about the 
three human resource priorities, most 
deputy ministers stated that they would be 
facing a significant recruitment challenge 
over the next five years. While they also 
expressed a general need for support from 
central agencies, we did not see any 
specific plans or requests for direction or 
additional resources. Some managers we 
interviewed mentioned that their 
department needed funding for interim 
positions and for training of new 
employees, yet this need does not seem to 
have been translated into specific planning 
demands. 


21.37 We asked the Treasury Board 
Secretariat if it had summarized the 
deputy ministers’ responses to the Clerk 
and assessed whether their plans were 
sufficient to meet the overall need for 
recruitment. The Secretariat had not done 
so because it was waiting for departments 
to submit individual business cases to 
address recruitment issues. It expects them 
to review the uses of resources, including 
the use of employees hired for a specific 
period and contractors before seeking 
additional resources for recruitment. The 
Secretariat needs to be more active in 
providing clearer direction to departments 
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on dealing with the recruitment priority 
and challenging their plans accordingly. 


21.38 Our review of the Secretariat’s 
documents indicates that operational 
planning and forecasting capacity varied 
significantly from one organization to the 
next. According to the Treasury Board’s 
recent estimates and projections, there is a 
need to increase recruitment levels 
substantially. However, these analyses do 
not break down the needed results by 
occupational groups. We did not find any 
evidence that the Board shared this 
information with the Commission 
(responsible for overall recruitment) or 
with departments (responsible for 
identifying their needs) to support the 
development of an overall recruitment 
strategy. 


21.39 The Public Service Commission 
could play a more proactive role in 
managing its Post-Secondary Recruitment 
program, given the recruitment priority, 
and in taking action where departmental 
responses seem to be inadequate to meet 
the recruitment challenge. 


21.40 ‘In our view, the current situation 
indicates that a government recruitment 
strategy is needed to provide direction for 
departments and the Commission on how 
best to manage the recruitment priority. 
We believe that if central agencies keep 
waiting for departments to act and 
departments keep waiting for direction 
and support from the central agencies to 
act, the recruitment priority will not be 
managed in a strategic and proactive 
manner. 


21.41 = In our April 2000 Report, 
Chapter 9, Streamlining the Human 
Resource Management Regime: A Study 
of Changing Roles and Responsibilities, 
we Stated that unresolved structural and 
systemic issues were partly the root of 
these problems. 


21.42 No one is clearly the driving 
force behind recruitment, and the 
government has not developed its strategic 
direction and goals for recruitment. For 
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Deputy ministers 
stated that they would 
be facing a significant 
recruitment challenge 
over the next 

five years. 
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We are concerned 
about the 
government’s ability 
to address the 
recruitment priority. 


For its part, Statistics 
Canada has formally 
made human resource 
planning a priority and 
recruited more 
employees to sustain 
its work force. 


these reasons, we are concerned about the 
government’s ability to address the 
recruitment priority. 


Planning Under the 
Post-Secondary Recruitment 
Program Is Ad Hoc 


21.43 We expected that recruitment 
levels for the Post-Secondary Recruitment 
program would be aligned with forecast 
vacancies according to retirement and 
attrition rates over five years. We also 
expected that based on established levels, 
recruitment strategies would be developed 
to reach and recruit the necessary 
candidates. 


21.44 Inreviewing some current 
profiles of senior managers, we learned 
that on average, it took them 10 years to 
move from their professional entry-level 
to an executive position. Given this fact 
and the objective of the Post-Secondary 
Recruitment program (to recruit for the 
short- and long-term needs of the public 
service), we found no analysis of the way 
the program will help to replace the many 
senior workers who will be eligible to 
retire in eight years or of its chance of 
success in doing so. 


21.45 Recruitment levels of the 
program are based on ad hoc requests of a 
few departments. We found no evidence 
that the Public Service Commission or the 
Treasury Board Secretariat challenge the 
number of positions proposed by 
departments to address the recruitment 
priority. 


21.46 Annual recruitment levels of 
some corporate development programs 
have been set, but have not been met. For 
example, over the last four years, the MTP 
has hired between 27 and 71 candidates. 
For the same period, recruitment levels for 
the MTP were set at 80 and then 100 for 
the last two years. The AETP has not 
reached its annual recruitment level of 14 
in the last four years. Though the FORD 
program does not have a set recruitment 


level, it has hired about 100 candidates 
annually. FORD officials say that the 
program has played a key role in renewing 
and rejuvenating the finance community. 
However, they recognize that more 
recruitment is needed because there is a 
growing need to implement a recruitment 
strategy for filling middle management 
positions in the financial community. 


21.47 For its part, Statistics Canada has 
formally made human resource planning a 
priority and recruited more employees to 
sustain its work force. Exhibit 21.3 
illustrates some key components of its 
recruitment practices. To meet its 
forecasts, Statistics Canada has recruited 
an average of 200 knowledge workers 
annually, using the Post-Secondary 
Recruitment program as the primary 
vehicle. 


21.48 In 1998-99 Fisheries and Oceans 
undertook a special recruitment initiative 
to renew their science and technology 
community and aimed to create 86 new 
positions in areas where gaps were 
predicted. The Department identified the 
Post-Secondary Recruitment program as 
the key to hire these professionals and 
scientists. The Department intended to 
convert 17 term employees to 
indeterminate (or permanent) status and 
recruit up to 69 candidates through the 
program. In fact, it reduced the target to 
81 new positions, converted the status 

of 53 of its term employees, staffed 

22 positions with temporary or casual 
employees and hired only six candidates 
through the program. 


21.49 In February 2000 Fisheries and 
Oceans stated in a letter to the Clerk that 
it would annually hire 100 more 
employees than its projected attrition rate. 
The department is not participating in this 
year’s fall campaign of the program and is 
still developing recruitment strategies and 
plans. For these reasons, we are concerned 
about the Department’s ability to meet its 
commitment in 2000-01. 
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Management support Exhibit 21.3 


¢« Management is continuously involved in human resource planning, including recruitment. ae 
Key Components of Statistics 
Canada’s Recruitment 
Practices 


| ¢ Policies, systems and practices are effective for managing work force needs because they have 
been tested and applied for about 10 years. 


e The Department follows a central approach to its recruitment and development programs and 
has a number of management committees and reporting structures that it uses to deliver them. 


Human resource planning 


e The Department clearly states the strategic direction. It integrates the strategic direction into 
the long-term planning process. It links the long-term human resource strategy to the strategic 
direction. 


e The Department integrates human resource planning into its operational planning. 


e The Department determines renewal requirements from long-term planning exercises and 
analyses. 


e The Department makes three- to five-year forecasts based on the need to deliver its programs 
and on attrition and promotion rates. 


e The Department continuously assesses the strength of its work force. 


Management of recruitment 


e The Department uses the Post-Secondary Recruitment program for many of its key 
| occupational groups. 


e The Department joins the Public Service Commission for promotional visits at university 
campuses. 


e Promotional material explains the Department’s working environment. 


e The Department stresses merit and fairness in its recruitment practices and provides sensitivity 
training to recruiters. 


e The Department shares information on its recruitment practices. 
e The Department’s candidates are offered structured training and development opportunities. 


e The Department maintains eligibility lists of qualified candidates. 


Performance assessment 


e The Department periodically assesses recruitment strategies. It conducts surveys to improve 
recruitment practices and determine whether it has met the expectations of candidates. 


e The Department gives feedback to the Public Service Commission on recruitment results. 
e The Department monitors recruitment costs. 


e The Department tracks acceptance and retention rates and uses this information to adjust future 
recruitment activities. 


Source: Statistics Canada 


21.50 The Department of Foreign 
Affairs and International Trade has hired 
an average of SO foreign service officers 
since 1997 and expects to hire 60 through 
the fall 2000 campaign of the program. In 
the past three years, it has hired up to 

15 management and consular affairs 
officers annually. Yet it is experiencing 
high attrition rates because almost a third 
of foreign service officers leave after 7 to 
12 years on the job. In addition, we noted 
that roughly 40 percent of officers and 

38 percent of executives will be eligible to 
retire in 2001. In our opinion, these 


demographics could leave the Department 
in a difficult position to replace staff if a 
majority of these employees decide to 
retire soon. Nevertheless, the Department 
is not further adjusting its recruitment 
levels under the program. 


21.51 Central agencies and most 
departments have not analysed labour 
markets to assess trends that affect the 
program. In our view, several factors will 
make it more difficult to attract candidates 
in the future: a smaller pool of candidates, 


Central agencies and 
most departments 
have not analysed 
labour markets to 
assess trends that 
affect the program. 
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increased competition for candidates, and 
workers’ changing expectations. 


21.52 To address the recruitment 
priority of the federal government, the 
Treasury Board Secretariat and the 
Public Service Commission should 
immediately develop a results-oriented 
strategy that will identify recruitment 
expectations for the government. 


21.53 In this strategy, the Public 
Service Commission, the Treasury 
Board Secretariat and departments 
should identify appropriate recruitment 
levels for the Post-Secondary 
Recruitment program. 


There Is a Trend to Hiring on a 
Term Basis 


21.54 Over the past 10 years, 
specific-period appointments have 
outnumbered indeterminate appointments 
by a wide margin (see Exhibit 21.4). 


21.55 Many managers we interviewed 
gave us a number of reasons for hiring 
employees on a term basis: uncertainty 
about long-term funding, short-term 
project work, and a quicker way to replace 
an indeterminate employee on assignment. 
They also prefer term hiring because it is 
faster than hiring for a permanent position 
and it gave them a chance to assess the 
suitability of the candidate — a “try 
before you buy” approach. (It should be 
noted that indeterminate employees have a 
probationary period of up to 12 months in 
some cases.) 


21.56 Managers also indicated that the 
conditions of the competitions for term 
employment are quite restricted in the 
eligible area of competition, specified 
skills, and the period to apply. These 
conditions limit the number of applicants. 
Managers reported that employees on term 
status are often converted to indeterminate 
status because they have a proven track 
record and have gained knowledge of the 
department and its operation. 


Exhibit 21.4 
Types of Appointments to the ae 
Public Service 
1998 
al Indeterminate 1997 
ee Determinate 1996 
1995 
48,300 
1994 Fa 
Note: Determinate eee ae 
1993 (TS) 
employment (term 
employment) includes 4,430 19,834 
part-time or full-time 02 
employment for a fixed, 4.443 21,860 
predetermined period. 199] a eee 
6,311 17,634 
L990 : é 
4,953 18,048 
Source: Public Service 1989 
Commission Annual | | | | 
Reports from 1989 to | : : 
1999 0 10,000 20,000 30,000 40,000 50,000 60,000 
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21.57 One director of a school of public 
administration told us that its students, 
who were studying for a career in the 
public service, were willing to accept a 
contract or a term appointment because 
they see this offer as the primary means of 
entry into the public service. The director 
noted that as a result, graduates would 
spend some of their energy on seeking 
indeterminate positions, rather than fully 
committing themselves to their work and 
dealing with the challenges of the job. 
They would also become cynical about the 
recruitment process and careers in the 
public service. Some managers we spoke 
to made the same observations. If hired on 
a term basis, employees would keep 
searching for a permanent position 
elsewhere and would likely leave the 
department if they received an offer. This 
continuous search has an impact on the 
work units’ productivity. 


21.58 Job creation and employment 
rates are rising and unemployment rates 
are falling; this trend means that there is 
more competition for fewer resources. In 
the 1990’s employment increased for 
knowledge workers in the management, 
scientific, professional and technical 
fields. University placement officers also 
indicated that qualified graduates might 
not consider the public service for 
short-term employment when competitors 
offer permanent positions. Consequently, 
the public service may be losing qualified 
candidates and may be finding it more 
difficult to attract them in an increasingly 
competitive market. 


21.59 In our view, this short-term hiring 
practice shows a lack of long-range 
planning. Managers are being reactive by 
filling their vacant positions with 
specific-period hiring, with little regard 
for long-range needs. This trend may also 
affect the planning and use of the 
Post-Secondary Recruitment program. 


21.60 The Public Service Commission 
should review departments’ hiring on a 
term basis and assess the extent to 
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which this practice has an impact on the 
government’s strategy for recruitment 
and on the use of the Post-Secondary 
Recruitment program. 


The Public Service Commission 
Needs to Be a Better Recruiter 


21.61 The Public Service Commission 
and departments share responsibilities for 
the Post-Secondary Recruitment 
program’s two campaigns that take place 
each year. 


21.62 As the overall manager of the 
program, the Commission is responsible 
for promotion, administration of the 
application process, initial testing and 
pre-screening. For corporate development 
programs (MTP, AETP, and FORD 
program), it often runs assessment centres 
and co-ordinates interviews. Once the 
Commission refers candidates to 
departments, its work is completed. 


Despite the lengthy 
recruitment process, 
Officials found that 
selected candidates 
were well qualified. 


21.63 Departments provide the 
Commission with the type and number of 
positions for which they will be recruiting 
and indicate the education and skills 
required. Some of them promote their 
positions on visits to university campuses. 
After the Commission has referred 
candidates, most departments undertake 
additional selection processes, including 
further review of applications, testing and 
interviews. In many cases, the process 
takes up to nine months from application 
to job offer. 


21.64 Despite the lengthy process to 
reach a final decision, officials from 
departments that used the program were 
very satisfied with the results and found 
that selected candidates were well 
qualified. 


In our view, short-term 
hiring shows a lack of 
long-range planning. 


Promoting the federal public service to 
university graduates could be improved 


21.65 We expected to find promotion of 
the public service and its values, and a 
realistic presentation of the working 
environment in general. 


EE 
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Students were not 
aware that the 
government was hiring 
and no longer 
downsizing. 


21.66 According to university 
placement officers and a recent survey of 
students, the Commission was not 
sufficiently and aggressively promoting 
the advantages and benefits of a career in 
the public service. Information on benefits 
and compensation, work environment, 
work challenges and careers was lacking. 
As well, students were not aware that the 
government was hiring and no longer 
downsizing. This fact is significant given 
the competition for qualified candidates. 


21.67 We noted that the Commission 
has undertaken some promotion 
initiatives. In 1999 the Commission 
prepared a Speaker’s Resource Kit, which 
contains general information on the 
federal public service. For the fall 2000 
campaign of the Post-Secondary 
Recruitment program, the Commission 
posted more information on its Web site 
(www.jobs.gc.ca) about the benefits of a 
career in the public service. 


21.68 In 1999 the Department of 
Foreign Affairs and International Trade 
produced documentation for candidates 
that presents a realistic picture of the 
advantages and disadvantages of a foreign 
service career. Statistics Canada has a 
booklet that outlines the selection process 
and the advantages of a career in the 
organization. 


21.69 In our opinion, more needs to be 
done to promote the public service as a 
career choice. 


Government visibility on campuses 
could be better co-ordinated 


21.70 We expected the Commission and 
departments to co-ordinate their visits to 
university Campuses. 


21.71 During the program’s campaigns, 
the Commission uses its regional offices’ 
staff to visit university campuses. Some 
departments that advertise positions 
through the program also go to 
universities; others do not. 


21.72 We noted that visits by the 
Commission and departments do not 
always coincide. Moreover, university 
placement officers indicated that students 
were sometimes confused about the image 
of the public service. 


21.73 Some students did not know if 
the Commission was recruiting for the 
public service, the Commission itself or 
the departments. Others were unaware that 
some departments belong to the public 
service. The Commission and departments 
need to review the promotion of the public 
service as a whole and explain more 
clearly how positions fit into the overall 
picture. 


21.74 University placement officers 
indicated that a united approach would 
have more impact. The Commission could 
promote careers in the public service in 
general, and departments could elaborate 
on the advantages of the opportunities 
they have to offer. The placement officers 
and human resource officials in private 
organizations believe that the most 
effective practice is to assign for every 
visit a senior manager who can answer 
broad questions and a recent recruit who 
can relate to students’ needs. 


21.75 University placement officers 
suggested that the government promote 
employment opportunities in the students’ 
first undergraduate years to encourage 
them to consider and prepare for a career 
in the public service. They also suggested 
that the government might wish to 
increase its visibility on campuses. This 
greater presence would allow it to further 
promote the public service as a career 
choice and provide more timely 
information to students on the 
Post-Secondary Recruitment program. 


The scheduling of the program’s 
campaigns raises problems 


21.76 University placement officers and 
government managers are concerned 
about the brevity and the restrictive timing 
of the program’s campaigns. Each 
campaign lasts some four weeks, usually 
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in September and again in January. 
Recruitment for corporate and 
departmental development programs is 
only done in the fall. The program’s fall 
campaign starts in the first week of the 
semester when most students select their 
courses, organize their timetables and 
settle in; the candidate’s assessment 
process takes place during mid-term 
exams. 


21.77 Information on the program’s 
Web site is limited, except during the two 
campaigns. There is no continuous 
application process for the Post-Secondary 
Recruitment program. 


The program does not offer 
opportunities across the country 


21.78 While over two thirds of the 
federal public service work outside the 
National Capital Region (NCR), we 
expected that the program would recruit 
for entry-level positions in other regions. 


21.79 We found that the program 
offered jobs primarily in the NCR. 
University placement officers told us that 
students are frustrated by the lack of 
regional work opportunities and the need 
to relocate to the NCR. The Commission’s 
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regional personnel believes that the 
program has to be tailored to regional 
recruitment needs because current 
opportunities in the NCR are not always 
attractive to candidates. 


21.80 Placement officers added that 
when the Commission’s regional 
representatives are on campus, they 
promote only positions advertised under 
the program. They do not talk about 
possible regional opportunities in 
departments that may arise during the year 
through general recruitment. 


21.81 Recruiters of organizations in the 
private and para-public sectors have more 
aggressive, timely and flexible 
recruitment practices (see Exhibit 21.5). 


21.82 When compared with the 
Commission’s recruitment practices, these 
organizations are more visible on 
campuses and regularly meet with 
placement staff and faculty members of 
institutions. They clearly define their 
corporate image and do not limit requests 
for applications to specific periods or 
campaigns. Their managers focus more on 
the recruitment results than on process, 
which is usually less time-consuming. 


21.83 The Public Service Commission 
has taken a number of steps to improve 


| Private organizations 

| e A Canadian financial institution participates 
in every job fair on university campuses and 
offers many perks to attract students. Its Web 
site is user friendly, and students can submit 
applications year-round. 


e A Canadian insurance company participates 
in every relevant fair on high school and 
university campuses that have an actuary 
sciences program. It offers summer positions 
to students for three years and eventually a 
permanent job. The company has a Web site 
that is user-friendly, and students can submit 
applications year-round. It also advertises 
job offers in all relevant newspapers and at 
Human Resources Development Canada. The 
company developed a hiring and recruiting 
kit for its managers. 


e An international high-tech firm pays for the 
tuition fees of the students it hires in the 
summer so that they can act as its 
ambassadors year-round on campus. In turn, 
the firm expects students to participate in 
career fairs and promote their employer. The 
firm’s Web site is user-friendly, and 
students can submit applications 
year-round. 


Para-public organization 


e A Canadian research establishment locates 
and tracks undergraduates in biological 
sciences and offers them summer or term 
positions. Its intention is to hire them when 
they graduate with a Ph.D. 


1 ae 
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There is no continuous 
application process for 
the Post-Secondary 
Recruitment program. 


Exhibit 21.5 


Recruitment Practices of 
Private and Para-Public 
Organizations 


Source: Canadian organizations 
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In 1998, nearly 5,600 
candidates met the 
basic requirements, 
but were not hired. 


With the call to renew 
the public service, we 
expected that more 
departments would 
have participated in 
the program. 


the performance of the Post-Secondary 
Recruitment program’s campaigns. 
However, university placement officers 
and government managers believe that 
more improvements are needed in 
promoting the public service as a career 
choice, increasing government visibility 
on campuses, and scheduling campaigns. 
In addition, expanding the program to 
offer more regional positions needs to be 
explored. 


21.84 To improve its performance as 
a recruiter of university students under 
the Post-Secondary Recruitment 
program, the Public Service 
Commission should review the design of 
the program to make it more flexible, 
adaptable, and timely. 


21.85 The Public Service Commission 
should better co-ordinate recruitment 
efforts with departments to promote the 
public service as a career choice and to 
offer opportunities across Canada. 


Greater Use Could Be Made of the 
Program’s Campaigns Results 


21.86 Some 11,500 graduates applied 
under the Post-Secondary Recruitment 
program’s 1998 campaign. The 
Commission referred some 6,400 of them 
to corporate development programs and to 
departments for further assessments. As a 
result, about 800 received appointments, 
leaving nearly 5,600 candidates who were 
interested in public service positions and 
met the basic requirements, but who were 
not hired. 


21.87 According to the Commission’s 
information system, general recruitment 
competitions in 1998-99 resulted in some 
7,000 appointments in the Administrative 
and Foreign Service and Scientific and 
Professional categories, of which some 
5,400 were for a term period. 


Inventory of qualified candidates would 
maximize results 


21.88 The Commission spends 
considerable time and money advertising 
positions, processing applications, and 
assessing candidates. However, except in 
very few circumstances, the Commission 
does not maintain an inventory of the 
program’s qualified candidates for 
referrals in other general recruitment 
processes. 


21.89 In our view, if government 
managers had access to inventories of 
pre-screened qualified candidates who had 
been assessed through a broad-based merit 
process, their staffing process could be 
more efficient. Managers might also be 
more encouraged to hire them on an 
indeterminate basis. 


21.90 The Commission manages an 
inventory of candidates for positions in 
information management and information 
technology (IM-IT) and uses innovative 
techniques to expand opportunities for 
graduates and to facilitate staffing for 
managers. The Post-Secondary 
Recruitment program could benefit from 
some of these techniques. 


Few departments use the program 


21.91 Between 1996 and 1999, nine 
departments regularly advertised positions 
in the Post-Secondary Recruitment 
program, and seven others did so 
occasionally. The many departments that 
do not hire using the program is puzzling, 
given that they have positions requiring 
workers with a university degree. 


21.92 Following the call for the renewal 
of the public service, we expected that 
more departments would have participated 
in the program. The number of them 
taking part in the fall 2000 campaign is 
higher than in previous years, yet it still 
represents a small portion of the federal 
government. We did not find any analysis 
of this lack of participation. 


21.93 The Public Service Commission 
should maximize the use of the 
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campaign results of the Post-Secondary 
Recruitment program by managing an 
inventory of pre-screened candidates. 
This inventory could be used to fill 
positions that are advertised during the 
program’s campaign and for any other 
government positions. 


21.94 The Public Service Commission 
should determine why more 
departments are not participating in the 
program and address departmental 
concerns in the program’s design. 


Assessment and Reporting of the 
Program’s Results Are Limited 


21.95 We expected the Public Service 
Commission would regularly assess 
activities of the Post-Secondary 
Recruitment program, particularly 
recruitment acceptance rates, retention 
rates and costs as well as selection tools. 


21.96 Private sector managers provide 
some analysis and feedback to the human 
resource divisions to adjust recruitment 
strategies (see Exhibit 21.6). 
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Assessment of the program’s 
recruitment activities is weak 


21.97 Recruitment acceptance and 
retention rates are not analysed. 

In 1999-2000 the Commission undertook 
the Longitudinal Study, which reviewed 
the recruitment process from the 
submission of applications to 
appointments for the fall 1998 campaign 
of the program. The objectives of the 


study were to improve the understanding In 1998, departments 
of the process, advise on the subject and filled only 62 percent 
identify any potential barriers to ay 
encouraging diversity in occupational of the positions they 
groups. had advertised 

21.98 The study states that departments through the 

filled only 62 percent of the positions they Post-Secondary 


had advertised through the Post-Secondary 
Recruitment program in 1998. The study 
reports a similar figure for previous years. 
The Commission acknowledged that it 
cannot explain these shortcomings 
because it has not systematically gathered 
information on the number of candidates 
who have qualified and have been selected 
through the program. For the same reason, 
it cannot advise the program’s regional 
co-ordinators to help them adjust their 
promotion efforts. To improve the 
monitoring of campaign results, the 


Recruitment program. 


Exhibit 21.6 


A Canadian insurance company expects feedback from managers to adjust its recruitment 
strategy. Managers are held accountable and report to the Chief Executive Officer if a new 
recruit does not succeed. The human resources division updates its hiring and recruiting kit 
based on feedback from managers. The division prepares an exit interview report, which 
includes the views of departing term employees, and submits it to the CEO for action. 


A Canadian financial institution conducts exit interviews of departing employees and adjusts its 
recruitment strategy accordingly. It also uses the results of exit interviews to update 
competencies required for entry-level positions. 


A Canadian chemical, mechanical and electrical company advertises work diversity to attract 
the right talent. It expects employees to have three to four careers with the company. After two 
years, the human resources division and managers discuss new avenues with employees. 
Because its turnover rate is very low, the company participates in an annual industrial survey on 
human resources. The survey provides information on the time to fill a position as well as 
recruitment acceptance rates and costs. 


An international industrial firm regularly assesses its recruitment practices and tackles 
competition from other firms for recruiting in the following way: “Continuous challenges have 
to be offered to employees. To reduce employees’ mobility, these challenges have to be better 
than those offered by competitors. Nowadays, competitive organizations’ recruitment strategies 
need to consider sculpting positions to attract the right talent.” 
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Examples of Feedback on 
Recruitment in Some Private 
Sector Organizations 


Source: Canadian organizations 
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The Public Service 
Commission has 
undertaken 

two separate studies 
to improve its 
performance in 
ensuring an 
employment equity 
perspective in the 
candidate’s 
assessment 
processes. 


Commission has established partnership 
agreements with participating departments 
that require them to report the reasons for 
the difference between the number of 
advertised positions and hired applicants. 


21.99 We have mentioned in 
paragraph 21.46 that some corporate 
development programs had difficulty in 
reaching their recruitment levels. These 
levels have not been revised based on past 
experience, and recruitment activities 
have not been revisited to meet these 
levels in the future. As well, we also 
found that there is no systematic tracking 
of candidates after their completion of 
these development programs to determine 
whether they have remained in the public 
service or moved into managerial 
positions. 


21.100 University placement officers 
would like to know the number of student 
applications that the Commission 
received, those referred to departments 
and corporate development programs and 
those appointed. This data would help 
them make recommendations on 
curriculum and counsel students on 
careers in the public service. 


21.101 We found that Statistics Canada 
tracks its recruitment acceptance and 
retention rates for decisions on human 
resource planning and adjusts its activities 
accordingly. The Department of Foreign 
Affairs and International Trade is looking 
at its problem in retaining foreign service 
officers. Subsequent to a recent 
departmental review of its results with the 
last campaign of the program, the 
Department is now streamlining its 
assessment process for Foreign Service 
and Management and Consular Affairs 
Officers programs because of similarities 
in basic recruitment requirements. 


21.102 Knowledge of recruitment costs 
is uneven. We expected that the 
Commission and departments identify 
recruitment costs and use the information 
to improve the management of the 
Post-Secondary Recruitment program. We 


found that knowledge and use of this 
information varied. 


21.103 The Commission has estimated 
that its direct costs for the program 
increased from $1.19 million in 1997-98 
to recruit some 350 candidates in one 
campaign to $ 1.38 millions in 1998-99 to 
recruit about 850 in two campaigns. It 
expected costs to reach $1.44 million 

in 1999-2000 to recruit 800 candidates. 
Assessment and selection costs incurred 
by departments and corporate 
development programs are not included in 
these figures. 


21.104 We found no cost-effectiveness 
analysis of the program’s components, 
such as visits to universities, various 
promotion activities and the use of 
regional offices. This analysis would 
allow the Commission to maximize its 
investment in these components and make 
necessary adjustments. 


21.105 Statistics Canada has a system to 
adequately monitor its recruitment costs. 
The Department of Foreign Affairs and 
International Trade recently began to 
budget and track its recruitment costs. 


21.106 In our view, the uneven 
knowledge of these costs for recruitment 
activities — a major investment in human 
resources — limits the ability of the 
Commission and the departments to 
monitor and improve the cost 
effectiveness of these activities. 


21.107 Selection tools have been 
assessed for employment equity bias. 
The Public Service Commission has 
undertaken two separate studies to 
improve its performance in ensuring an 
employment equity perspective in the 
candidate’s assessment processes. The 
first study is the External Review of 
Selected Instruments of the PSC’s 
Personnel Psychology Centre (Cronshaw 
Report). It assessed a sample of selection 
tools and tests for potential bias or adverse 
impact, including some used in the 
Post-Secondary Recruitment program. 
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21.108 Asa result, the Commission 
replaced two tests used for the program 
because of their adverse impact on 
employment equity. The Department of 
Foreign Affairs and International Trade 
changed the education requirements for 
foreign service officers from a master’s 
degree to a bachelor’s degree. Statistics 
Canada made the same change. 


21.109 The second study, the 
Longitudinal Study discussed earlier, 
focussed on the program’s outcomes for 
equity groups. 


21.110 The Task Force on Visible 
Minorities, chaired by Lewis Perinbam, 
suggested in June 2000 that one member 
of a visible minority be recruited for every 
five new recruits. The task force reported 
its concern about the significant 
reductions between the number of 
applications received from visible 
minorities and the number of those 
appointed. It suggested that the merit 
principle was not being meaningfully 
applied. The Commission’s Longitudinal 
Study indicated that the program’s 1998 
campaign resulted in 22 percent of 
appointments for visible minorities, well 
above their labour market participation 
rate and equal to the hiring ratio proposed 
by the task force. However, the study also 
noted a considerable decline in 
representation from applications to 
referrals, and then to appointments. 


21.111 During our audit, university 
placement officers mentioned the need for 
the Commission to increase its contact 
with various on-campus student 
associations involved in employment 
equity issues to further facilitate 
recruitment under the program. We are 
aware that the Commission has increased 
its outreach activities, but it may need to 
keep universities better informed about its 
activities. 


21.112 We expect the Commission to use 
the Longitudinal Study to improve and 
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monitor the results of its future program 
campaigns. 


21.113 The program’s efficiency and 
effectiveness has not been assessed. 
Under the program, the departments hired 
over 1,500 candidates between 1996-97 
and 1998-99, and about an expected 800 
in 1999-2000. We did not find any 
assessment to determine the extent to 
which recruitment levels under the 
program are adequate to address the 
government’s recruitment priority. We did 
not find any analysis to determine whether 
the program complemented other 
recruitment activities in the public service. 


21.114 Because the overall expectations 
for recruitment have not been stated, it is 
difficult to assess the efficiency and 
effectiveness of the program. For the same 
reason, it is difficult to determine the 
extent to which the program is 
contributing to the renewal of the 
government’s professional and 
management work force and whether 
corrective action is necessary. 


Reporting of results needs major 
improvement 


21.115 We expected the Commission to 
report the results of the Post-Secondary 
Recruitment program. 


21.116 We reviewed the Commission’s 
annual reports, its reports on plans and 
priorities and performance reports. We 
found that the information presented in the 
annual reports is usually limited to 
quantitative data on numbers and types of 
appointments. These documents do not 
provide long-term trend analysis or 
qualitative information such as hiring 
trends of recruitment programs, expected 
shortages in occupational groups, and 
participation and performance of 
departments in hiring activities by status 
of employment. Given that the 
Commission has not stated expectations 
for recruitment levels, it cannot 
adequately report results. However, the 
Commission needs to at least explain the 
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difficulties that it has encountered in 
trying to meet recruitment levels and 
discuss the lessons that it has learned. 


21.117 Departments and corporate 
development programs do not report 
systematically to the Commission the 
results of their activities through the 
program against their expectations. Nor do 
they provide feedback on the 
Commission’s level of service. This 
information would help to assess the 
extent to which changes should be made 
to the recruitment strategy. 


21.118 The Public Service Commission 
and departments should systematically 
assess recruitment activities under the 
Post-Secondary Recruitment program 
to improve its efficiency and 
effectiveness. The Commission and 
departments should also report to 
Parliament on the success of the 
program. 


The work of the Committee of Senior 
Officials on Recruitment is not complete 


21.119 To highlight the importance of 
the government’s recruitment priority, the 
Clerk of the Privy Council created in 
October 1999 a subcommittee of deputy 
ministers on recruitment. The objectives 
were to develop a problem statement, a 
federal strategy, and a work plan to 
address government recruitment needs. 
The Committee of Senior Officials 
(COSO) asked the subcommittee to ensure 
that recommendations consider options 
for, among other things, the strategic use 
of recruitment, future government 
recruitment campaigns and the integration 
of recruitment activities with the broader 
exercise of examining program integrity 
(ensuring that existing programs are 
adequately structured and resourced). 


21.120 The following statement from the 
report reflects our findings: 


Demographic and labour market 
realities are leaving [the federal 
public service] with gaps to be filled, 


and there will be even greater gaps 
ahead unless [the government] takes 
serious, sustained action on 
recruitment. The public service faces 
an even greater recruitment 
imperative than most organizations in 
Canada — all of this taking place in 
an increasingly competitive labour 
market for skilled [knowledge] 
workers. [Moreover, in the next] 

five years, a significant number of 
executives and more importantly, 
their feeder groups, will be eligible to 
retire. Current recruitment and career 
development processes are not 
enough to meet the expected demand. 


21.121 Based on our review of the report 
and its proposed action plan, we believe 
that the plan needs further development in 
setting the government’s recruitment 
levels, building human resource capacity, 
implementing recruitment alternatives 
such as “overstocking” or maintaining a 
pool of generic positions, and reporting 
results. 


21.122 For the Post-Secondary 
Recruitment program, the subcommittee 
suggests a number of initiatives that 
mirror our recommendations: improving 
the promotion of the program, increasing 
the government’s visibility on university 
campuses, managing an inventory of 
pre-screened qualified candidates, and 
encouraging more departments to use the 
program. However, the subcommittee’s 
report and action plan do not state specific 
results expected from these initiatives. 


Conclusion 


21.123 Our audit and many public 
reports we have reviewed indicate that the 
federal public service is experiencing 
shortages of knowledge workers, which 
are expected to worsen as demographics 
and the retirement of the baby boomers 
come into play in five years. This looming 
work force crisis will leave the public 
service with professional and management 
positions that will be difficult to fill in an 
increasingly competitive environment. 
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Therefore, the government must take 
immediate action. 


21.124 Current efforts to deal with the 
recruitment of human resources are not 
well co-ordinated because leadership of a 
government recruitment strategy has not 
been assigned. Moreover, these efforts are 
neither timely nor efficient and, as such, 
are not likely to be effective to renew the 
professional and management work force 
of the public service in five years. 


21.125 The Public Service Commission, 
in its current role of recruiter, succeeds in 
attracting qualified knowledge workers, 
but this success is not translated into the 
hiring of sufficient numbers. The design 
and management of the Post-Secondary 
Recruitment program needs to be 
enhanced. The Commission also needs to 
maximize the use of the program. In 
particular, the Commission needs to 
assume its assigned responsibilities as the 
recruiter: 


* to encourage the participation of 
departments under the program; 


* to ensure that government 
recruitment levels are spelled out and 
meet the government’s objective; and 


* to better promote the public service 
as a career choice to university graduates 
across Canada. 


21.126 The Treasury Board, in its role of 
employer, needs to provide guidance and 
guidelines to the human resource 
community and government managers to 
help them adequately deal with the 
recruitment issue. Furthermore, the Board 
needs to anticipate departments’ resource 
requirements to help them manage the 
issue. 


21.127 Departments need to review their 
participation in corporate initiatives and 
current hiring practices and carefully 
assess the impact of these activities on 
their long-term human resource planning. 
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21.128 In addition, the Commission, the 
Treasury Board Secretariat, and 
departments need to review how they can 
best inform Parliament of their progress in 
recruiting indeterminate employees. A 
significant improvement would be to 
report quantitative and qualitative 
information, such as recruitment practices, 
resource allocations and achieved results 
against recruitment levels and specific 
areas of shortages by occupational groups. 


21.129 In our opinion, even if immediate 
actions are taken to improve recruitment 
practices under the Post-Secondary 
Recruitment program, shortages of 
knowledge workers may nevertheless 
occur. The Commission may need to 
consider other alternatives for recruitment 
above the entry level to minimize 
potential consequences for the public 
service in delivering programs and 
providing services of quality to 
Canadians. 


Response of the Treasury Board 
Secretariat for the government and the 
departments audited: The seriousness of 
recruitment to the public service is 
demonstrated by the government’s 
commitment in the 1999 Speech from the 
Throne and by the Clerk of the Privy 
Council who created and chaired a 
Deputy Minister Committee mandated to 
develop action strategies on this issue. 
This committee also recognized (consistent 
with the Auditor General’s observations) 
that tailored recruitment strategies would 
be required and that entry into the public 
service should be a consideration at all 
levels, not only our perceived entry level 
positions of the past. For example, the 
Department of Foreign Affairs and 
International Trade has broadened its 
recruitment strategy to include mid-career 
transfers and secondments as well as the 
traditional entry-level recruitment. 


The Treasury Board is also strengthening 
its role in overseeing and planning the 
human resource needs of the public 
service. As one element of assistance, the 
Treasury Board Secretariat has launched a 
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Web site on Demographic Analysis of the 
Federal Public Service Workforce. This 
site contains references and links to 
research, both inside and outside of the 
public service. The Secretariat will be 
working more closely with the Public 
Service Commission and departments. Our 
objective is to work in collaboration in the 
analysis, planning and development of 
recruitment strategies that will meet 
department’s unique needs. 


As well, deputy heads are leading renewal 
activities. Fisheries and Oceans has 
created several working groups at the 
assistant deputy minister level to lead 
workplace improvement initiatives. These 
initiatives focus on ways we can enhance 
our capability to develop meaningful 
human resource plans that can address 
recruitment and retention strategies. 


The government concurs with the Auditor 
General’s recommendations. We agree that 
the public service must adopt an active 
renewal approach —one that uses this 
opportunity to enhance our representation 
of the citizens we serve and that attracts 
the people with the skills we need for our 
future work. 


Response of the Public Service 
Commission: The Commission agrees with 
the recommendations made on the 
Post-Secondary Recruitment (PSR) 
program and the audit observations are 
consistent with the findings from the 
Commission’s studies, which support the 
improvements that are under way and 
those that are being planned. 


The PSR program has been very important 
in attracting and recruiting new 
employees to the public service. The 
Commission agrees with the statement 
made in one of the main points of the 
audit report to the effect that the Public 
Service Commission is recruiting qualified 
candidates. In the fiscal year 1999-2000, 
there were 16,636 PSR applicants for 
1,131 advertised positions, demonstrating 
a good market penetration on campuses. 
However, this is not the only mechanism 


for hiring managers. In 1998-99, 
approximately 85 percent of all 
indeterminate appointments were made by 
other means than PSR. 


The Commission requires significant new 
investment in resources to maximize the 
results of the PSR program campaign and 
make it a strategic recruitment tool for 
entry into the public service. The 
Commission needs the resources to rebuild 
capacity that was lost as a result of the 
Program Review in 1994 and the 
consequent decrease in recruitment 
activities. Recruitment demand has picked 
up significantly in the last two year. The 
Commission has re-allocated its resources 
and: 


¢ established an additional annual 
winter PSR campaign since 1998; 


¢ introduced on-line job information 
and application through www.jobs. gc.ca; 


* enhanced promotion and marketing 
efforts through increased campus visits; 
increased presence in career fairs and 
expanded outreach to employment equity 
groups; 


* undertaken specialized recruitment 
strategies for groups such as the 
information technology community; and 


* re-introduced inventories for high 
demand areas such as Computer Systems 
Administration (CS) and Personnel 
Administration (PE) occupational groups. 


Specific comments related to the 
recommendations: The Commission will 
work collaboratively with the Treasury 
Board Secretariat to support departmental 
efforts in the elaboration of a 
results-oriented recruitment action plan 
and in the determination of appropriate 
recruitment levels through their human 
resource planning exercises. The 
Commission will continue its efforts to 
assist departments with modeling and 
demographic analysis. Twelve 
departments, eight regional areas and four 
functional communities (policy, 
communications, regulatory inspection, 
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science and technology) have benefited 
from these services recently. The 
Commission is also exploring with the 
Secretariat the establishment, in 2001, of 
a joint centre of expertise in demographics 
and modeling. 


The President of the Public Service 
Commission has increased the PSR 
program ’s visibility by personally writing 
to his colleague deputy ministers, asking 
them to participate in the program’s 
campaigns, to designate recruitment 
“champions” and to undertake 
partnership agreements with the 
Commission on departmental targets and 
results framework. It has also established 
an interdepartmental advisory committee 
on recruitment programs. 


The Commission, in collaboration with the 
Secretariat, has produced a recruitment 
action plan. Under this initiative, the 
Commission has submitted a business case 
for increased resources to modernize and 
rebuild recruitment capacity across the 
public service, which includes the 
redesign of the Post-Secondary 
Recruitment program’s campaign. With 
increased resources, the Commission will 
launch a new PSR by fall 2001. The 
redesign will address, among others: 


* establishment of a planning 
framework and results-oriented strategy 
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that identifies recruitment expectations 
and the appropriate PSR recruitment 
levels, including regional opportunities, 
and proactive communication/marketing 
to departments seeking their commitment 
to the use of the PSR program; 


* assessment of term usage and its 
impact on the program’s strategy; 


* development of public service-wide 
brand to promote the public service as a 
career and specific marketing and 
promotion activities for target areas or 
occupational groups; co-ordination of 
recruitment efforts and government 
visibility on campus; establishment of 
inventories; 


* investment in technology to provide 
tools to hiring managers and human 
resource advisors and streamline the 
recruitment process; and 


* development of a performance 
measurement framework and tools. 


The PSR program redesign will include an 
appropriate performance measurement 
framework and reporting requirements. 
The Commission will report to Parliament 
through its Performance Report and 
Annual Report. 
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% About the Audit 


Objective 


The purpose of our audit was to determine the extent to which the Post-Secondary Recruitment program will 
enable the government to renew its professional and management work force and to assess the effectiveness 
and efficiency of the program. 


Scope 


We focussed on how the Public Service Commission of Canada and the Treasury Board Secretariat centrally 
administer recruitment activities. We examined how the Department of Foreign Affairs and International 
Trade, Fisheries and Oceans and Statistics Canada identified their human resource requirements and made use 
of the program. In particular, we examined the Management Trainee Program, Foreign Service Development 
Program, Financial Officer Recruitment and Development program, and the Accelerated Economist Training 
Program. 


Criteria 
Our audit criteria are based on the best practices of public and private sectors in recruitment and selection. 


We expected the following: 


¢ Renewal requirements that are stated and are consistent with the government’s strategic direction. 


° Decisions to develop or acquire human resources that are co-ordinated in a timely manner between 
central agencies and the senior officials of departments. 


e Analyses that are conducted over a five-year forecasted period with respect to 
- the current and available professional and management work force and its feeder groups; 


- future renewal requirements of the professional and management work force and its feeder groups, 
including competencies; and 


- work force gaps by demographic cohort (difference between requirements and available work force). 


Effective and efficient recruitment strategies that are targeted to fill short- and long-term renewal 
requirements. 


Recruitment strategies that reflect corporate public service staffing values (e.g., competency, 


non-partisanship and a representative work force) and provide a realistic view of the working 
environment in general. 


Policies, systems and management practices that are in place. Roles, responsibilities and accountabilities 


that are identified for recruitment and selection decisions. Related recruitment costs that are identified 
and used in decision making for recruitment. 


Best recruitment practices that are shared to facilitate recruitment performance improvement. 
Analyses of acceptance and retention rates that are conducted and used to improve recruitment strategies. 


Selection and assessment tools that are analyzed for biases and barriers to ensure that objectivity and 
transparency are maintained throughout the selection process. 
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¢ Regular assessment that is done of recruitment strategies and programs against expected forecast 
requirements and related costs. 


* Reporting that is conducted on recruitment strategy, programs and results against expectations. 


Audit Team 


Assistant Auditor General: Maria Barrados 
Principal: Kathryn Elliott 


Claude Brunette 
Denise Coudry-Batalla 
Esther Fine 

Ernest Glaude 
Catherine Gohier 
Gerry Rosinski 

Robyn Roy 


For information, please contact Kathryn Elliott. 
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Appendix A 


List of Occupational Groups for the Scientific and Professional, and 


Administrative and Foreign Service Categories 


Occupational groups for the Giese 


and Professional category 


Actuarial Science 

Agriculture 

Architecture and Town Planning 
Auditing 

Biological Sciences 

Chemistry 

Dentistry 

Defence Scientific Service 
Economics, Sociology and Statistics 
Education 

Engineering and Land Survey 
Forestry 

Historical Research 

Home Economics 

Law 

Library Science 

Mathematics 

Medicine 

Meteorology 

Nutrition and Dietetics 

Nursing 

Occupational and Physical Therapy 
Pharmacy 

Physical Sciences 

Psychology 

Scientific Research 

Scientific Regulation and Patent 
Social Work 

University Teaching 

Veterinary Medicine 


Administrative Services 
Commerce 

Computer Systems Administration 
Financial Administration 

Foreign Service 

Information Services 
Management Trainee 
Organization and Methods 
Personnel Administration 


Purchasing and Supply 
Program Administration 
Translation 

Welfare Programs 


Source: Treasury Board Secretariat’; Employment Statistics for the Federal Public Service 
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Appendix B 
Summary of the Report of COSO’s Subcommittee on Recruitment, July 2000 


Most departments believe that they are facing a significant recruitment challenge over the next five years, as they 
prepare for the retirement of an aging work force. Some departments are conducting leading-edge recruitment, and 
others know they have to act. Departments need to play a more corporate role in sharing innovative approaches to 
develop more strategic recruitment practices for the future. 


While departments expect to make increasing use of the Post-Secondary Recruitment program, the program accounts 
for only one percent of new appointments each year in the professional categories. The Management Trainee Program 
and other corporate initiatives exist and are used, but it is not clear that these are linked to concrete, strategic human 
resources plans. 


The subcommittee recommended that public service leaders do the following: 


* offer permanent jobs, where appropriate, to qualified new recruits and assess their potential during the 
probationary period, instead of using term assignments to determine their suitability; 


¢ make human resource planning an integral part of business planning and align recruitment with future 
business direction; 


¢ inform the President of the Public Service Commission of any barriers to recruitment that need to be 
eliminated; and 


¢ challenge departmental corporate services to support innovative approaches. 


The subcommittee also recommended that human resource specialists do the following: 


* ensure that recruitment managers have the support to effectively recruit, orient and retain new employees; 
and 


* provide strategic advice and guidance on how to recruit the right people for the right place at the right 
time. 


The subcommittee mentioned that the Public Service Commission and the Treasury Board Secretariat have made an 
important contribution to moving forward on recruitment, and they are committed to going even further. The central 
agencies submitted an action plan, Meeting the Recruitment Challenge. The action plan sets out recruitment priorities, 
including the following: 

* There will be more use of the Post-Secondary Recruitment program and attention to current occupational 
shortages. 
* A pool of generic positions will be created into which young professionals can be hired. 


* Workplace-of-choice efforts will centre on promoting the public service among potential recruits, 
especially on university campuses. 


* Strategic planning will help to clarify recruitment needs for the next three to five years, with specific 
business cases for key recruitment challenges. Over time, this initiative will lead to recruitment planning 
as an integral part of doing business for all managers. 


Here are some of the short-term deliverables from the central agencies’ action plan: 
° For strategic planning, recruitment forecasts will be submitted by April 2001 and April 2002. 
¢ For market analysis, research and review to support forecasting/demographic modelling and work force 
studies will be ongoing in 2001 and 2002. 
* For information on progress, results and accountability, a Post-Secondary Recruitment Partnership 
Agreement will be signed with departments in 2001 and 2002. 
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On February 2000 the Clerk of the Privy Council requested deputy ministers to provide information on their 
department’s plan for recruitment. The following reflects some main points of the executive summary: 


¢ It is not clear from the replies received that the sense of urgency has reached the level of hiring managers. 


* Most departments are in the planning phase of recruitment and are developing recruitment strategies. 
There is little information on the specifics of what is planned. 


¢ There is less clarity on what needs to be done for general recruitment across the public service. 


¢ Several departments signalled the need to move to “overstocking” (recruiting more employees than 
actually needed to prevent major recruitment in the future) for a specified period of time to bring in and 
develop successors and help address the workload. 


° A one-size-fits-all employment package will not meet the diverse needs of functional communities and 
regions. 


¢ Some departments are experienced recruiters and would be a source of practical expertise. 
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Chapter 22 


Development of the Universal 
Classification Standard 


A Follow-up 


The follow-up work was conducted in accordance with the legislative mandate, 
policies and practices of the Office of the Auditor General. These policies and 
practices embrace the standards recommended by the Canadian Institute of 


Chartered Accountants. 
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Funding process did not follow normal processes for projects of this size 
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An uneven consultation process 
We advised the Secretariat of our concerns in 1998 


Improving the Standard’s Design 
The Secretariat followed an exhaustive method of data collection 
The new information was used to improve the Standard’s design 
Validating the Standard was the weak link 
The 5,000-position sample used for validation had serious limitations 
The Secretariat guides and methodologies reflect the state of the art 
Conditions in departments were not conducive to successful validation 
Results of the evaluation of the 5,000-position sample were not reliable 
The Standard benefited from a second validation 


Improving the Application of the Standard 
Additional quality assurance measures after validation 
Departments took seriously the need to improve the quality of the position evaluation results 
Departments call the Universal Classification Standard a promising job evaluation tool 
The Secretariat is taking further steps to ensure the Standard’s adequate application 
An opportunity for the Secretariat to demonstrate to what extent the Standard achieves 
its objectives 


Weighting the Standard’s Elements 
A preliminary weighting scheme was developed 


The Challenges Ahead 
Converting the existing classification of positions to the new classification system will 
be a critical challenge 
Conditions for success 


Conclusion 
About the Follow-up 
Exhibits 


22.1 Major Phases and Key Activities of the Universal Classification Standard 
Pata Factors and Elements of the Universal Classification Standard 2.0 
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Development of the Universal 
Classification Standard 


A Follow-up 


Main Points 


22.1 The Treasury Board Secretariat has made major progress in implementing our 1996 audit 
recommendation on the development of the Universal Classification Standard for the federal public service. This 
project is fundamental to the reform of human resource management, particularly for staffing and compensation. It 
is a monumental and highly complex undertaking that requires much time. The present system is outdated and no 
longer meets the requirements of today’s public service and tomorrow’s. 


22.2 In the last few years, the Secretariat has designed a standard intended to measure the content of diverse 
types of work in the public service, and has subjected it to a series of tests in departments. Since 1998, 
departments have been writing new work descriptions for all positions and evaluating them against the Standard. 
In the course of this project, the Treasury Board Secretariat has developed and issued guides and methodologies 
based on best practices in job evaluation. 


22.3 Despite these achievements, the government still has to address significant issues: 


¢ The Treasury Board Secretariat needs to ensure that the new Standard is being applied appropriately 
before fully converting to it. 


* Departments need to ensure that their work descriptions are accurate and their position evaluation 
results valid and adequate. 


¢ The Secretariat needs to perform further testing, based on accurate and up-to-date evaluation data, to 
establish the extent to which the Standard meets its stated objectives to be universal and 
gender-neutral. 


* The government needs to ensure that reasonable and realistic financial resources are available to 
convert to the new Standard. 


Background and other observations 


22.4 A cclassification standard is a measuring tool for determining the relative value of positions in an 
organization. The Universal Classification Standard, now being developed, must be capable of evaluating all 
positions in the public service on the basis of work content. The results will be used to establish a hierarchy of 
positions and the compensation that reflects their requirements. 


22.5 The Standard will apply to all Public Service employees for whom the Treasury Board is the employer, 
with the exception of the executive group. At August 2000, there were some 150,000 positions in 72 separate 
occupational groups, in some 65 departments and agencies subject to the new classification system. The 
development of a new classification standard has to meet the requirements of the Canadian Human Rights Act on 
equal pay for work of equal value. 


22.6 The existing classification standards do not reflect today’s emphasis on serving the public and managing 
for results. They also do not make reference to the fact that public servants work more and more in partnership 
with other levels of government and the private sector. 
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22.7. The development of a new classification standard was launched in 1991 as part of the Public Service 
2000 initiative. In 1996, we reported serious deficiencies (May 1996 Report, Chapter 5). Pursuant to our 
recommendations and the findings of its own review, the government began to improve the Universal 
Classification Standard. This follow-up chapter is a progress report on the initiative to date. 


The Treasury Board Secretariat has responded positively to our findings and recommendations and has 
affirmed that progress will continue to meet the Universal Classification Standard objectives. 
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Introduction 


22.8 With the launch in 1989 of its 
Public Service 2000 initiative, the 
government was seeking to renew the 
public service and achieve a culture 
oriented more toward serving clients and 
delivering results. It set up a task force to 
review classification and job evaluation. 
The task force recommended in 1990 that 
the government develop a new job 
evaluation system that would be simple, 
universal and gender-neutral. By 

fall 1993, a new job evaluation system 
developed by the Treasury Board 
Secretariat was in “final draft” form. We 
decided that before the new system was 
fully implemented, we would examine 
whether it would enable the Secretariat to 
achieve its objectives. The Secretariat’s 
work continued, and the June 1995 
version of the Universal Classification 
Standard became the subject of the audit 
we reported on in 1996. 


22.9 The 1996 audit focussed on 
assessing whether the proposed Standard 
would indeed be universal, gender-neutral, 
simple and cost-neutral. We concluded 
that while the Standard at that time was a 
significant improvement, that it would be 
universal and gender-neutral was not 
assured. Furthermore, writing and 
evaluating job descriptions remained 
relatively complex and costly. 


22.10 Since 1996, the Secretariat has 
focussed its efforts on achieving 
universality and gender neutrality in the 
design of the Standard and ensuring that 
this new evaluation tool is valid and 
reliable. 


Why is a new classification standard 
needed? 


22.11 Largely designed almost 40 years 
ago, the current classification system is 
outdated and no longer meets the 
operational requirements of today’s public 
service and tomorrow’s. A number of the 
classification standards no longer reflect 
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the nature of the work employees do or 
the way they do it. And no classification 
standard at present reflects today’s 
emphasis on serving the public and 
managing for results. Public servants work 
more and more in partnership with other 
levels of government and the private 
sector, and existing standards make no 
reference to the requirements of that work. 


22.12 Since 1991, when the project to 
develop a new standard began, the 
introduction of new technologies has 
changed the nature of work at all levels. 
Today, over half of public servants are 
“knowledge worker” professionals. 


22.13 As the 1990 PS 2000 task force 
report acknowledged, existing job 
evaluation systems are too complex, take 
too long and cost too much to operate. 


22.14 As the federal employer, the 
Treasury Board is subject to the Canadian 
Human Rights Act. The task force pointed 
out in 1990 that existing job evaluation 
systems did not meet the requirements of 
the Act, and it recommended the 
development of a new system that would. 
Moreover, decisions on major pay equity 
complaints in the public service heard by 
different Canadian Human Rights 
Tribunals concluded that the current job 
evaluation systems did not meet the 
guidelines on equal pay for work of equal 
value. In one decision, the Tribunal 
directed the Treasury Board to correct 
inequities in the job evaluation system 
with a new classification standard, free of 
gender bias. In another decision, the 
Tribunal dealt only with the salary 
component of the complaint; it deferred 
its decision on the gender discrimination 
component, in part because the 
government indicated that it was 
developing a single, universal 
classification standard that would be 
gender-neutral. 


22.15 The present classification system 
confines employees to the narrow limits of 
their job descriptions and is a considerable 
obstacle to redeployment of resources. Its 
rigidity prevents the public service from 
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The Universal 
Classification 
Standard is critical to 
reforming human 
resource management 
in the public service. 


Developing a new 
standard that 
adequately takes into 
account the value of 
all significant 
characteristics of work 
in the public service is 
a huge task. 


adapting to the changing needs of 
programs and their clients. 


22.16 The Universal Classification 
Standard is critical to reforming human 
resource management in the public 
service. Without a classification standard 
that can serve as a strong basis for 
staffing, promotion and compensation, the 
employer will find it hard or impossible to 
attract and retain the best people and 
ensure that a competent public service 
continues. 


A very large and highly complex project 
that requires time 


22.17 The process of developing a 
classification standard or a job evaluation 
system is not a mathematical exercise. It 
involves a degree of subjectivity and the 
use of judgment. To minimize the 
subjectivity and to obtain valid, reliable 
and credible job evaluation results, the 
process must therefore be rigorous, 
ordered and systematic. Furthermore, to 
ensure that the system will meet the needs 
of the organization and that the results 
will be consistent, acceptable and 
equitable, the development process needs 
to include all stakeholders: management, 
employees, employee representatives and 
regulatory bodies. Working out this 
process and the quality control 
mechanisms to support it requires a great 
deal of time and is essential to the success 
of such a project. 


22.18 A classification standard is a 
measuring tool for determining the 
relative value of positions within an 
organization. The Universal Classification 
Standard must be able to evaluate all 
positions in the public service on the basis 
of their work content. When all the 
positions have been evaluated, the results 
can be used to establish a hierarchy of 
positions and the salaries that reflect their 
requirements. This approach to 
determining salaries was suggested by the 
PS 2000 working group on compensation 


with the aim of ensuring internal equity in 
the organization. 


22.19 The size of the organization is 
another factor that adds considerably to 
the complexity, the risks and the time 
involved in developing a new standard. 
The Standard will apply to all federal 
Public Service employees (Public Service 
Staff Relations Act, Schedule 1, Part 1), 
for whom Treasury Board is the employer, 
with the exception of the executive (EX) 
group. The more jobs there are and the 
more varied their work content, the more 
complicated and risky the process of 
developing a standard will be. In the 
public service, for example, the Position 
Classification Information System showed 
in August 2000 some 150,000 positions in 
72 separate occupational groups, in some 
65 departments and agencies that are 
subject to the new classification system. 
The variety of positions is vast, ranging 
from research scientists, nurses, program 
managers and policy analysts to ships’ 
officers, electricians, secretaries, pay 
clerks, and maintenance workers. 
Developing a new standard that 
adequately takes into account the value of 
all significant characteristics of work in 
the public service is a huge task. 


22.20 Meeting the requirements of the 
Canadian Human Rights Act is another 
significant challenge. Often ignored, the 
requirements are based on the concept of 
equal pay for work of equal value. In 
developing a job evaluation system in a 
federal organization, the value of positions 
in predominantly female groups must be 
compared with that of positions in 
predominantly male groups, using a 
common standard and taking into account 
the four factors specified in section 11 of 
the Act: responsibilities, skills, effort and 
working conditions. The government must 
comply with its own legislation. In 
addition, a Canadian Human Rights 
Tribunal has decided that a gender-neutral 
job evaluation system is required by the 
Act. Given that requirement and the 
Tribunal’s decision, the government 
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decided to develop a single universal 
standard capable of comparing positions 
across different groups. 


22.21 Gaining stakeholder participation 
and acceptance at each stage would prove 
to be a complex and time-consuming 
endeavour for the Treasury Board 
Secretariat. The project involved 

16 different unions, each representing the 
specific interest of a different part of the 
public service. 


22.22 Compensation rates, market 
forces, and supply and demand of certain 
skill sets outside the public service may 
also affect the ability to reach a consensus 
and may not always reflect internal equity. 
The Treasury Board Secretariat must work 
to find an acceptable solution that takes 
into account pay equity requirements, 
internal equity, market forces, and 
affordability — difficult challenges to 
reconcile. 


Focus of the follow-up 


22.23 This chapter discusses our 
follow-up of the audit we reported in 

May 1996, Chapter 5. We assessed the 
extent to which the Treasury Board 
Secretariat has implemented our 
recommendation dealing with the 
development and validation of the 
Standard. The chapter examines the 
development of the Standard, its important 
components, the methodologies developed 
by the Secretariat, and the application of 
the Standard in departments and agencies 
up to September 2000. 


22.24 Our follow-up work does not 
cover the other recommendations we 
made in 1996, on the issue of cost and on 
the simplification of the job evaluation 
system and the occupational group 
structure. We note, though, that in 
replacing 70-odd existing standards, the 
new single Standard represents a 
considerable simplification of the 
classification process. 
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22.25 In the process of developing the 
Standard, since 1996 the Secretariat has 
sought the advice of the Canadian Human 
Rights Commission on the application of 
the Canadian Human Rights Act. The 
Canadian Human Rights Commission has 
also been investigating the extent to which 
the Standard is gender-neutral. We 
therefore excluded the Standard’s gender 
neutrality objective from the scope of our 
chapter, relying instead on the 
Commission’s work and opinion. 


22.26 This chapter focusses mainly on 
the validity, reliability and universality of 
the Standard; its testing and application 
and the development of supplementary 
tools were still under way as we were 
writing. Further details on the objectives 
and scope of our work are presented at the 
end of this chapter, in About the 
Follow-up. 


Observations and 
Recommendations 


22.27 The Treasury Board Secretariat 
has made major progress in implementing 
our 1996 audit recommendation on the 
development of the Universal 
Classification Standard for the federal 
public service. Despite problems along the 
way, the Secretariat has set up a process 
for developing the Standard that is 
ordered, systematic, and significantly 
more exhaustive than was the process 
prior to 1996. The major phases of the 
project are outlined in Exhibit 22.1. 


Some Challenges Impacted the 
Project Throughout 


Changes in leadership and staff during 
the project brought some constraints 


22.28 Since 1996, the project has 

seen successive changes in leadership: 
two Secretaries of the Treasury Board, 
three Chief Human Resources Officers, 
and two Assistant Secretaries in charge of 
the project. At other levels, staffing has 
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Exhibit 22.1 


Major Phases and Key 
Activities of the Universal 
Classification Standard 


Source: Treasury Board 
Secretariat documentation 


¢ Work information gathering: November 1996 to December 1997 


- Samples of 737 positions 
Design development and informal testing of the Standard: January to August 1998 


¢ First validation of the Standard version 1.0: July to December 1998 


- Sample: 5,000 positions across 56 departments 
- Writing and evaluating job descriptions of these 5,000 positions in departments 
- Qualitative and quantitative analyses 
¢ Revision of the Standard version 1.0 and release of the Standard version 1.1: January to 
April 1999 
¢ Second validation of the Standard version 1.1: April to July 1999 
- Evaluation of the same sample of 5,000 work descriptions (some revised) in a controlled 
environment 
- Qualitative and quantitative analyses 
° Revision of the Standard version 1.1 and release of the Standard version 2.0: July to 
October 1999 
¢ Application of the Standard to all positions in the public service: ongoing since August 1999 


¢ Development of an analytical framework to improve quality: January and February 2000 


¢ Workshops with departments to ensure the consistent application of the Standard: February to 
May 2000 


¢ Release of supplementary technical bulletins and guides: July 2000 
° Audit of work descriptions of 1,000 positions: September 2000 — ongoing 
* Qualitative assessment of application of 1,000 positions: October 2000 — ongoing 


e Periodic assessment of the Standard’s application: to come 
its 


° Development of weighting tools and scenarios: June 1999 to December 1999 


¢ Release of preliminary weights: February 2000 
° Impact analyses of the preliminary weights: ongoing 


° Conversion planning 


¢ Development and implementation of communication and change management strategies 
* Compensation policy and pay structure 

* Collective agreements 

° Conversion 
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been largely by secondment, with 
significant turnover on the core team. 
With changing staff came different views 
on what constituted a reasonable and 
satisfactory standard. This added 
complexity to the project’s management 
and led to numerous changes in the 
schedule. 


Funding process did not follow normal 
processes for projects of this size 


22.29 The funding process followed for 
the development of the Standard did not 
sufficiently take into account the size, 
complexity and importance of such a 
project. As with other major projects like 
Modernization of Comptrollership and the 
Financial Information Strategy, we 
expected that adequate funds would be 
granted to departments and the Secretariat 
to properly deliver this project. The 
Treasury Board Secretariat established a 
“loan” fund of some $60 million to help 
departments with the costs of training, 
writing work descriptions, and evaluating 
jobs. However, departments that used this 
funding have to pay it back within three 
years. Departments have largely had to 
fund the project from their existing 
budgets and resources. Since the project 
has taken longer than anticipated and 
involved significant time and effort of line 
managers and employees as well as 
human resources staff, its financial impact 
to date has likely been underestimated and 
will be difficult to account for accurately. 


22.30 In our opinion, the approach to 
funding this project has represented an 
ongoing risk to its successful completion. 
The scope of our follow-up did not allow 
for auditing the full costs of developing 
and validating the Standard and applying 
it in departments. However, the Secretariat 
has told us that the direct incremental cost 
of the organizational unit at the Secretariat 
who developed the Standard has been 
about $20 million since 1996. Some 

40 departments of various sizes have 
reported costs that total some $35 million 
up to June 2000. 
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Timelines that proved to be unrealistic 
have compromised progress 


22.31 From the start, the Treasury 
Board Secretariat imposed on itself a 
schedule that greatly underestimated the 
nature and complexities of a project like 
the development of the new Standard, 
which it planned to have ready by 

31 March 1999. The Canadian Human 
Rights Commission has expressed concern 
to the Secretariat since early 1998 about 
the pace at which aspects of the Standard 
have been developed and applied. 


22.32 Throughout the project, and as 
we were writing this chapter, the tight 
deadlines have caused departments major 
difficulties in evaluating positions, as they 
tried to avoid compromising the quality of 
the results while meeting the deadlines. 


An uneven consultation process 


22.33 One of the difficulties in 
developing the Standard has been the 
varying degrees to which stakeholders 
have been consulted, particularly 
employee unions and the Canadian 
Human Rights Commission. From 

late 1995 until early 1998, the Treasury 
Board Secretariat held meetings with all 
stakeholder groups together, and their 
participation and co-operation made it 
possible to consider their concerns and 
advance the project effectively. In 
mid-1998, however, when the initial 
design was complete and the project was 
moving to the next phase, the Secretariat 
considered that this process was no longer 
effective and it reduced the level of 
consultation to separate meetings with 
each stakeholder group. The Secretariat 
has stated that the same topics were 
covered in its meetings with every 
stakeholder group. However, we believe 
that the change to separate meetings left 
each group with no confidence that it had 
a grasp on all facets of the project and 
unaware of the concerns of other 
stakeholder groups. It also left stakeholder 
groups with questions and concerns about 
the transparency of decisions. Since 


In our opinion, the 
approach to funding 
this project has 
represented an 
ongoing risk to its 
successful 
completion. 


Given the risks and the 
potential implications 
for the Standard’s 
chances of success, 
we told the Secretariat 
in October 1998 that 
we were concerned 
about the project’s 
tight timelines and the 
reduced level of 
consultation. 
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In 1996-97, the 
Secretariat established 
a rigorous method 

and several quality 
control mechanisms 
for collecting 
comprehensive, valid 
and reliable 
information on a wide 
range of job content. 


mid-1999, however, consultations have 
improved and have been more open. 


We advised the Secretariat of our 
concerns in 1998 


22.34 Géiven the risks and the potential 
implications for the stakeholders’ 
co-operation, the project’s viability, and 
the Standard’s chances of success, we 
formally expressed our concerns to the 
Secretariat in October 1998 about, for 
instance, the project’s tight timelines and 
the reduced level of consultation. The 
Secretariat responded that it believed the 
change in consultation was appropriate, 
given the stage the project had entered, 
but indicated that it recognized the need 
for subsequent consultation. In our view, 
the Secretariat’s decisions seemed to be 
guided mainly by its desire to meet the 
31 March 1999 deadline for converting all 
positions to the new Standard. As 
experience later demonstrated, the 
timelines were unrealistic, and full 
consultation was critical for the project to 
proceed smoothly. 


Improving the Standard’s Design 


The Secretariat followed an exhaustive 
method of data collection 


22.35 To ensure that the Standard could 
measure the value of all work done in the 
public service, the Secretariat needed a 
very good understanding of the breadth of 
that work. This understanding was critical 
to determining the elements the Standard 
would need to be universal, and the scales 
needed to measure their value. The 
Secretariat designed two samples of 
positions to collect information about the 
current work in the public service, based 
on criteria intended to reflect the wide 
diversity of positions and to take into 
account the requirements of the Canadian 
Human Rights Act. 


22.36 The two samples comprised a 
total of 737 positions that, according to 
the Secretariat, reflected a combination of 


key jobs and positions of various types. 
The Secretariat considered that they 
provided a good illustration of the variety 
of work in the public service while also 
taking into account positions held 
predominantly by males and those held 
predominantly by females. 


22.37 The final composition of these 
two samples resulted in part from a 
consensus negotiated among various 
stakeholders. This approach to defining 
the samples, while reflecting the 
Secretariat’s desire to meet stakeholders’ 
expectations, was an approach based on 
judgment and we would expect the 
Secretariat to have taken into account the 
risks of using a sample arrived at in this 
way. While the sample covered a wide 
range of jobs, the documentation we 
reviewed did not show to what extent it 
covered the full range, or which jobs were 
left out. Given that its design was based 
on these samples, we did not have 
sufficient assurance to conclude that the 
new Standard could capture and evaluate 
all job content in the public service. In 
response to our concerns, the Secretariat 
told us it would remain diligent and would 
capture the missing job characteristics, if 
any, during the next stages of the 
Standard’s development. 


22.38 In 1996-97, the Secretariat 
established a rigorous method and several 
quality control mechanisms for collecting 
comprehensive, valid and reliable 
information on a wide range of job 
content. The data collection method was 
tested first in a pilot project to verify the 
effectiveness of the collection tool itself, 
the collection process, and the training kit 
for those who would be collecting the 
data. 


22.39 The Secretariat interviewed the 
incumbents of 737 positions in two 
samples, at 114 work sites across Canada. 
During this phase, the Secretariat set up 
quality control mechanisms to ensure that 
the information supplied by the 
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incumbents was up-to-date and complete, 
and reflected the actual work. 


The new information was used to 
improve the Standard’s design 


22.40 Working with departments, 
unions, and consultants who were 
specialists in job evaluation, the 
Secretariat conducted a critical review of 
all components of the version of the 
Standard we had audited in 1996. It 
continued to improve the Standard’s 
design, using the results of its review and 
the analysis of the data collected on job 
content. Several versions of the Standard 
were produced, each with successive 
in-depth modifications to both its design 
and its components. 


22.41 Between February and 

mid-May 1998, the Secretariat conducted 
informal tests of the modified Standard in 
a small sample of about 250 jobs in seven 
departments that had volunteered to 
participate. The Secretariat used the 
results as a basis for a number of changes. 
It also decided that it was time to 
undertake a formal validation in a sample 
of positions throughout the public service. 


Validating the Standard was the 
weak link 


22.42 The objective of validating the 
Universal Classification Standard was to 
verify its validity, reliability, universality 
and gender neutrality when applied to a 
representative sample of jobs in the public 
service. Given the scope and complexity 
of the task and its impact on the success of 
the project, we looked for indications of 
the level of accuracy, reliability and 
validity in the job evaluation results that 
departments and agencies provided to the 
Secretariat to validate the Standard. We 
expected that the data used for validation 
purposes were up-to-date and met the 
requirements of the Secretariat’s 
guidelines. 
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The 5,000-position sample used for 
validation had serious limitations 


22.43 We expected that the Secretariat 
would have developed a methodological 
framework that described the assumptions 
underlying the sampling, recognized the 
limits of the sample and, where necessary, 
determined the impact of those limits on 
the analysis of the job evaluation results. 
We saw only limited documentation of the 
sampling approach. The Secretariat could 
provide information on the size, general 
characteristics and purposes of the sample 
but not on the assumptions used in 
sampling and their impact. 


22.44 = In March 1999, following its 
assessment of the appropriateness of the 
sampling method, Statistics Canada 
concluded that the sampling method was 
essentially based on judgment and had 
produced a sample that could not be 
shown to be representative of the full 
diversity of jobs in the public service; and 
that analyses based on such a sample 
could be misleading. Some officials of the 
Secretariat concurred. 


22.45 The sample contained some 
2,700 different jobs, accounting for some 
5,000 sampled positions. The most 
common and key jobs were selected in 
large numbers. This, in effect, limited the 
diversity of the positions in the sample, 
resulting in reduced assurance that it 
represented the full range of work 
characteristics in the public service. We 
agree with Statistics Canada that the 
results of validation based on such a 
sample cannot be extrapolated in a 
statistically reliable way to the current 
public service positions. At a minimum, 
decision makers should understand the 
limitations of these samples in their 
deliberations. 


The Secretariat guides and 
methodologies reflect the state of the art 


22.46 The May 1998 Standard, Version 
1.0, was the first version validated. To 
ensure that departments would apply its 
Standard as consistently and precisely as 


Several versions of the 
Standard were 
produced, each with 
successive in-depth 
modifications to both 
its design and its 
components. 


Statistics Canada 
indicated that the 
results of validation of 
the selected sample of 
positions could not be 
generalized to all 
current public service 
positions. 
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The Secretariat gave 
departments and 
agencies a project 
management 
framework that 
identified critical 
success factors and 
performance 
indicators. 


The Secretariat 
counted on a sound 
validation process in 
departments to 
guarantee adequate 
results. 


possible, the Secretariat provided a 
number of tools for them to work with the 
Standard and become familiar with it. For 
example, it developed methodology 
guides to enable departments to write their 
work descriptions and evaluate them in 
accordance with the intent of the elements 
in the Standard. It also developed a 
methodology for developing model work 
descriptions to speed up the exercise and 
provide for consistent descriptions and 
evaluations of similar or identical 
positions across departments. These tools 
are based on best practices and reflect the 
state of the art in job evaluation. 


22.47. The Secretariat gave departments 
and agencies a project management 
framework that identified critical success 
factors and performance indicators. It also 
provided sources of information from 
which both the departments and the 
Secretariat could verify the way they 
carried out and managed the exercise. 


22.48 The Secretariat also set up an 
organizational infrastructure to give 
departments the necessary support for 
applying the Standard and to follow the 
progress of the job evaluation exercise. 
Training and information sessions were 
held for a number of participants. In order 
to obtain valid and reliable data, the 
Secretariat made it compulsory for 
departments and agencies to apply its 
methodologies and procedures in the 
validation. 


Conditions in departments were not 
conducive to successful validation 


22.49 We visited eight departments and 
examined the way they evaluated their 
positions from March 2000 to the end of 
September 2000. Our interviews and our 
review of documentation revealed the 
following: 


* In some departments we visited, 
senior management gave hesitant support 
to the application of the new job 
evaluation system because of the 
prevailing scepticism about the viability 
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of the project and the new values 
embodied in the Standard. 


- Most of the departments we visited 
were not ready to apply the Standard 
during 1998. For instance, some 
departments and agencies had not 
reviewed the classification of their 
positions to make sure it was valid before 
using the Standard. Nor had they checked 
the accuracy of their classification data 
banks and identified all generic jobs to 
reduce the workload in writing job 
descriptions. As a result, some 
departments had difficulty applying the 
Standard while at the same time making 
sure the classification status of their 
positions was current. 


¢ The timetable imposed on 
departments caused them serious problems 
and compromised the quality of the data 
bank on which the Secretariat based its 
analyses: 


- Many departments were unable 
to offer all the necessary training 
in writing work descriptions and 
evaluating positions. 


- In many cases, the work 
descriptions were not reviewed 
by supervisors and employees to 
ensure that they were accurate 
and complete. 


- Most departments we visited 
were unable to review the 
position evaluation results 
systematically for consistency 
and accuracy before sending 
them to the Secretariat. 


- A number of departments 
deplored the fact that the 
timetable underestimated the 
learning curve needed to master 
the Standard. 


22.50 Testing a new job evaluation plan 
of this scope in so many organizations 
requires a structured, well-defined 
framework. The Secretariat provided such 
a framework that outlined expected 
results. In reviewing the progress reports, 
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we noted that departments described the 
activities, processes and procedures they 
had set up but not the extent to which they 
achieved the desired results. The 
Secretariat counted on a sound validation 
process in departments to guarantee 
adequate results. In our opinion, the 
Secretariat needed to take additional 
monitoring steps to ensure that position 
evaluation results were reliable. 


Results of the evaluation of the 
5,000-position sample were not reliable 


22.51 We learned in our interviews 
conducted from March to the end of 
September 2000 that, to the best of their 
knowledge, departments generally had 
applied the Secretariat’s directives and 
methodologies in writing work 
descriptions and evaluating jobs. 
However, evaluation committees in 
several of the departments we visited 
reported that some job descriptions they 
had evaluated were incomplete or of poor 
quality. All departments we visited 
indicated that the results of the 5,000 
positions they sent to the Secretariat could 
not be considered reliable or accurate and 
required extensive review. 


22.52 In December 1998 the Secretariat 
conducted detailed analyses of several 
aspects of the Standard, without full 
knowledge of the poor quality of data 
from departments. In our opinion, this first 
validation exercise had important 
limitations and could not be used to 
demonstrate adequately that the Standard 
was universal. However, we observed that 
the Secretariat used the results of these 
analyses to improve the reliability and 
validity of several elements in Version 1.0 
of the Standard. 


22.53 The Secretariat introduced 
corrective measures in January 1999, 
targeting the aspects of the Standard that 
presented the greatest difficulties. It 
significantly revised seven elements of the 
Standard and produced supplementary 
guidance on its application. A new version 
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of the Standard, Version 1.1, was 
published in March 1999. 


The Standard benefited from a second 
validation 


22.54 It is considered best practice to 
undertake several validations in order to 
obtain sound results. The Secretariat 
carried out a second validation to verify 
whether the changes incorporated in 
Version 1.1 of the Standard had improved 
its reliability. That is, could the Standard 
be interpreted and applied consistently in 
a number of different situations across the 
public service? 


22.55 The second validation was 
deliberately conducted in a central and 
controlled environment, with the 
participation of 250 managers and 
employees selected for their sound 
knowledge and their experience in using 
the Standard. Experts on the Standard and 
on gender neutrality were available on-site 
to provide clarification, when necessary. 
While the profile of the people selected 
and the support provided were appropriate 
to test the validity of the Standard — 
whether the Standard measures what it is 
supposed to measure — it was not 
adequate for testing the Standard’s 
reliability. This type of ideal environment, 
not representative of the normal 
environment of the public service, would 
likely remove factors that could 
negatively affect the reliability of the 
Standard and could skew the validation 
results. 


22.56 The second validation used the 
same 2,700 jobs accounting for the 
5,000-position sample with work 
descriptions that had had a number of 
deficiencies. Although departments said 
they had the opportunity during the 
second validation to revise those work 
descriptions, all incumbents and their 
managers could not verify their accuracy 
and completeness. In addition, the 
departments themselves were still learning 
the Standard. 


22.57 The Secretariat performed a 
thorough qualitative and quantitative 


All departments we 
visited indicated that 
the results they sent to 
the Secretariat for the 
5,000-position sample 
could not be 
considered reliable or 
accurate and required 
extensive review. 
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Some departments 
were concerned about 
the consistency and 
reliability of 
interdepartmental 
ratings. 


In our opinion, based 
on the work the 
Secretariat had done, 
it should have more 
strongly qualified its 
statement and not 
concluded that the 
Standard met the 
objectives. 


The Secretariat 
developed a rigorous 
and exhaustive model 
for analyzing the 
evaluation results, and 
it undertook activities 
to improve the quality 
of application. 


analysis of the revised evaluation results, 
using outside consultants, to assess the 
universality of the Standard Version 1.1. 
This led to added minor improvements, 
and Version 2.0 was released 

(Exhibit 22.2). 


22.58 In March 1999, the Secretariat 
considered that the Standard was not 
perfect but was “good enough” to be 
released, given that most of the planned 
“functionality” had been tested and 
proved to be working. In August 1999, the 
Secretariat said it had strong evidence to 
indicate that the Standard had achieved its 
objectives of universality and gender 
neutrality. In our opinion the Secretariat 
could not demonstrate this convincingly, 
given the weaknesses of the 
5,000-position sample, the poor quality of 
the data and work descriptions, and the 
“ideal test bed” in which the second 
validation was conducted. In our opinion, 
based on the work the Secretariat had 
done, it should have more strongly 
qualified its statement and not concluded 
that the Standard met the objectives. 


22.59 In September 2000, the 
Secretariat did qualify the results of the 
validation more prudently. It indicated 
that the Standard was a vast improvement 
over previous Classification systems and 
that it had significantly improved 
reliability while reinforcing the design 


goals of universality and gender neutrality. 


Improving the Application of 
the Standard 


Additional quality assurance measures 
after validation 


22.60 In reviewing position evaluation 
results submitted up to January 2000, the 
Secretariat and departments noted several 
potential anomalies. The Secretariat 
developed a rigorous and exhaustive 
model for analyzing the position 
evaluation results, and it undertook 
activities to improve the quality of 
application of the Standard. The goal was 
to diagnose the causes of the potential 


anomalies and decide on appropriate 
corrective action. 


22.61 According to the Secretariat’s 
records, about half of the departments and 
three unions participated in workshops 
offered by the Secretariat and designed to 
review some 30 occupational groups. 
Following the workshops, the Secretariat 
issued further technical guidance to clarify 
the Standard and provide for its consistent 
interpretation. In our opinion, the 
Secretariat needs to consider expanding 
this process to the remaining occupational 
groups, given that the objective is 
universality. 


Departments took seriously the need to 
improve the quality of the position 
evaluation results 


22.62 Our interviews with departments, 
our review of the files showing how the 
Standard had been applied after the 
evaluation of the 5,000-position sample, 
and the responses to our short 
questionnaire administered in 

September 2000 showed increased efforts 
to pursue and improve the application of 
the Standard. For example: 


¢ In most cases the training activities 
were continued. 


* Most departments told us they had 
applied the methodologies developed by 
the Secretariat to write and evaluate work 
descriptions. Some said they had 
developed other tools, adapted to their 
own needs, that had not necessarily been 
validated by the Secretariat. But they 
considered that in doing so they had 
observed the same principles. 


+ Although the level of mastery of the 
Standard has improved, interpreting it 
consistently in departments is a constant 
challenge because of the high turnover 
among members of departmental 
evaluation committees. 


« All departments told us that because 
of the many changes to the Standard and 
the anomalies that surfaced, they have 
carried out a critical review of the work 


Report of the Auditor General of Canada — December 2000 


Development of the Universal Classification Standard: A Follow-up 


Exhibit 22.2 


Factors and Elements of the Universal Classification Standard 2.0 


Elements: 


1. Information for the Use of Others measures the extent and impact of responsibility for information that is used by others. 


2. Well-being of Individuals measures direct responsibility for supporting the health and development of individuals and for 
helping them to function better in their environment and in Canadian society. This responsibility includes protecting them from 
harm and providing for their welfare. 


3. Leadership of Human Resources measures responsibility for leading people who are working to achieve the goals of the Public 
Service of Canada. 


4. Money measures responsibility in the work for the stewardship and comptrollership of financial resources. The responsibility 
falls into three areas: planning and controlling, acquiring funds, and spending funds. The element values these responsibilities 
according to the worker’s latitude to take action and make decisions within the federal policies, guidelines, and regulations that 
govern the management of funds. 


5. Physical Assets and Products measures direct responsibility for the custody, use, production, maintenance, repair, protection, 
and disposal of physical assets used, and products created, in doing the work. It also measures responsibility for making 
arrangements for facilities, equipment, and materials for conferences and events. Both purpose and impact are measured. 


6. Ensuring Compliance measures the responsibility in the work for making decisions about compliance with government-accepted 
standards, guidelines, regulations, and legislation and for taking appropriate action to foster and enforce such compliance. 


Elements: 


7. Job Content Knowledge Application measures the depth and breadth of the knowledge required to perform the work. Knowledge 
includes concepts, methods, practices, principles, procedures, processes, techniques, and theories. Knowledge can be acquired 
through training, education, or experience, or it can be the result of a natural ability. 


8. Contextual Knowledge measures the degree to which the work requires knowledge of people, organizations, external 
circumstances, and legislation and regulations. 


_ 9. Communication measures the skills required in the work to convey and receive messages. Both communication in (receiving) 
and communication out (conveying) are measured. 


10. Motor and Sensory Skills measures the proficiency that the work requires in controlling body movements and in using the senses 
to make distinctions. 
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Exhibit 22.2 (cont'd) 


Effort Factor measures the mental and physical exertion required by the work. 


Elements: 


measured. 


11. Intellectual Effort measures the mental exertion associated with solving the problems encountered in the work. Both intensity of 
effort and constraints on resolving problems are measured. 


12. Sustained Attention measures the effort required by the work to focus one or more of the senses. It also measures the sensory effort 
to remain attentive, sometimes to things that do not appear to be changing. Both intensity of effort and degree of distraction are 


13. Psychological/Emotional Effort measures the mental exertion required to cope with psychologically demanding work. Both 
intensity of effort and degree of control are measured. 


14. Physical Effort measures the amount of physical exertion required by the work. It recognises the effort involved in both movement 
and stillness. Both the intensity of effort and duration Gees of wee are measured. 


Working Conditions Factor measures the physical and psyehoogea conditions unde ; 
potential effects on the health of employees. 


Elements: 


15. Work Environment measures the exposure to disagreeable psychological and physical work environments. Both the psychological 
environment and the physical environment of the work are evaluated. 


16. Risk to Health measures the exposure to unavoidable mental and/or physical risks or hazards to health resulting directly from the 
performance of work. This element captures only those risks remaining after the application of normal safety precautions. 


Source: Universal Classification Standard, UCS 2.0, Factors and Elements, Treasury Board of Canada, Secretariat, October 1999. 


descriptions written and the position 
evaluation results since validation began. 


* Departments took into account the 
supplementary guidance that the 
Secretariat provided for interpreting the 


Standard. In response to our questionnaire, 


all departments said they have used the 
tools the Secretariat provided between 
February and August 2000, even though 
they arrived late in the review process. 
According to departmental 
representatives, in September 2000 there 
was still a lot of work to be done to 
integrate the changes required by the 
supplementary guides. 


* In September 2000, the majority of 
departments we visited indicated varying 
degrees of satisfaction with the quality of 
their work descriptions and the accuracy 
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of the evaluation of positions they 
consider finalized. The others preferred 
not to express an opinion but described the 
measures they have put in place to ensure 
sound results. Some departments were 
concerned about other issues, such as the 
consistency and reliability of 
interdepartmental ratings. 


+ All but two of the departments we 
visited believed that the quality control 
mechanisms they have put in place over 
recent months provide them with a high 
level of assurance that the position 
evaluation results they consider finalized 
are valid and reliable. 
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Departments call the Universal 
Classification Standard a promising job 
evaluation tool 


22.63 We asked each of the 
departments we visited to indicate, based 
on the knowledge and experience they 
acquired in using the Standard, if this job 
evaluation tool had enabled them to value 
all positions. The six departments that 
responded directly to this question replied 
in the affirmative. They said that the 
Standard can properly value the full range 
of work in the department. The other two 
departments made no direct assessment. 
Some departments expressed concerns 
about the value of some specific jobs 
under the Standard and the compensation 
offered in the external labour market. 


The Secretariat is taking further steps 
to ensure the Standard’s adequate 
application 


22.64 ‘In our interviews with all 
stakeholders, including the Canadian 
Human Rights Commission, and in 
comments received from some unions, a 
concern has emerged about the ability of 
the Treasury Board Secretariat to 
demonstrate that departments have 
adequate quality control mechanisms in 
place and have applied the Standard 
appropriately and produced valid position 
evaluation results. 


22.65 As we were writing this chapter, 
the Secretariat had already started 
initiatives to ensure the Standard’s 
adequate application. For instance, the 
Secretariat commissioned an audit of work 
descriptions of 1,000 positions. The audit 
is intended to examine the apparent 
degree to which departments have adhered 
to the Secretariat guidelines and also to 
assess the position content validity of the 
work descriptions. In addition, the 
Secretariat was developing a methodology 
for a qualitative assessment of the 
Standard’s application; this was in the 
preliminary stages. This is aimed at 
examining the consistency and 
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appropriateness of the Standard’s 
application within particular occupational 
groups. 


22.66 In our opinion, it is critical for 
the Secretariat to have confidence in the 
quality of position evaluation results with 
the Standard, for all occupational groups. 


22.67 The Secretariat is also planning 
to perform periodic assessments of the 
Standard’s application in departments. 
These assessments are intended to 
identify, through quantitative analyses, 
indicators of the adequacy or potential 
anomalies of the position evaluation 
results before the conversion. The 
Secretariat is planning to conduct further 
qualitative analyses based on these 
indicators. 


An opportunity for the Secretariat to 
demonstrate to what extent the 
Standard achieves its objectives 


22.68 The Secretariat’s initiatives we 
have described are necessary and relevant 
to fill the gaps identified by the two 
validation exercises. In our opinion, the 
Secretariat has a great opportunity to 
conduct robust testing and to demonstrate 
convincingly to stakeholder groups the 
extent to which the Standard meets its 
objectives. 


Weighting the Standard’s 
Elements 


A preliminary weighting scheme was 
developed 


22.69 In fall 1999, while the 
departments were evaluating positions, the 
Secretariat was developing the weighting 
scheme for the Standard. 


22.70 Weighting is the percentage 
assigned to each element in a 
classification standard to establish the 
relative importance of all the elements in 
order to reflect the organization’s 
objectives and values. These values must 
be free of gender bias in order to comply 


The Secretariat 
commissioned an 
audit of the work 
descriptions for 
1,000 positions. 


The Secretariat has a 
great opportunity to 
conduct robust testing 
and to demonstrate 
convincingly to 
stakeholder groups 
the extent to which the 
Standard meets its 
objectives. 
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Converting the 
existing classification 
of positions to the new 
system will inevitably 
cause disruption, no 
matter how perfect the 
new system may be. 


with the requirements of the Canadian 
Human Rights Act. 


22.71 Given the sensitive and 
subjective nature of the exercise, to 
establish the weighting the Secretariat set 
up an ordered and systematic process that 
enabled it to obtain the views of all the 
stakeholders, from various perspectives. 
For example, it created a forum for a 
group of current and former senior 
officials and employees where they could 
discuss the importance of each element in 
the Standard and then to establish the 
hierarchy of elements that they thought 
best reflected public service values. The 
design team also produced a ranking 
based on the intent of the Standard. 


22.72 Based on interim data, the 
Secretariat then developed a number of 
scenarios for weighting the elements of 
the Standard to determine the hierarchy 
that would be most acceptable to all 
stakeholders. Initially, criteria for this 
decision included several variables, 
among them the requirements of the 
Canadian Human Rights Act, the impact 
of the position evaluation results on the 
organizational structure, and the degree of 
organizational disruption implied. 


22.73 Through consultation with 
departments, unions and the Canadian 
Human Rights Commission, the 
Secretariat was trying to achieve a 
balanced scheme that would assign a 
value to all work in the public service, 
unlike the weighting scheme proposed 
in 1996. 


22.74 The Secretariat recognizes that 
before approving a final weighting 
scheme, it must conduct the necessary 
analyses using current data that are 
reliable and accurate. It will do so once 
the departments have finished evaluating 
all their jobs. We would support this 
course of action. 


The Challenges Ahead 


Converting the existing classification of 
positions to the new classification 
system will be a critical challenge 


22.75 Position classifications will be 
converted to the new classification system 
from an imperfect, outdated system 
designed on the basis of values and 
principles of work organization that are 
almost 40 years old. 


22.76 Conversion means that all 
employees in the future will be able to 
easily compare the value of their 
positions, which they could not always do 
before. However, comparing the value of 
positions under the new Standard with 
their value in the existing system could 
produce misleading results. No one has a 
precise idea of the magnitude of errors 
and anomalies in the present classification 
of positions. The last internal audit of 
classification was performed over 10 years 
ago. Even then, many jobs were found to 
have been misclassified. 


22.77 Converting the existing 
classification of positions to the new 
system will inevitably cause disruption, no 
matter how perfect the new system may 
be, and this is even more predictable in an 
organization the size of the public service. 
The Standard will introduce new values, 
and the results will lead to changes in the 
organizational hierarchy. 


22.78 The conversion will involve costs 
that are difficult to determine now with 
any certainty. A number of issues that will 
affect the costs of conversion still need to 
be resolved, such as negotiations with 
unions and establishment of the pay 
structure. At present, we do not know if 
the Secretariat has a compensation policy 
that defines the relative importance of 
internal equity, external equity, individual 
or group performance pay or other pay 
incentives and defines the relationships 
among them. Establishing a compensation 
policy from the perspective of strategic 
human resource management will be a 
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critical and essential element in 
successfully implementing the new 
system. 


Conditions for success 


22.79 A lot of work has been done so 
far in departments to implement the 
Standard in order to convert their jobs to 
it. At the end of our follow-up in 
September 2000, we asked the eight 
departments we visited, and some unions, 
what they believed could be the 
Standard’s key conditions for success. 
According to them, success would depend 
on conditions such as the following: 


+ due regard given to the results of the 
gender neutrality assessment of the 
Standard conducted by the Canadian 
Human Rights Commission, and the 
update and consolidation of all the tools 
developed by the Secretariat or specific 
occupational communities to ensure that 
the Standard is complete, free of gender 
bias and universal, and provides for 
consistent and adequate interpretation of 
all elements of the Standard; 


* consultation and agreement with 
departments on the timelines needed to 
ensure that the work is done well. Several 
departments and unions believe they need 
more time, given the nature and the 
impact of such a project on the whole 
public service. Many indicated that there 
is still a lot of work to be done to ensure 
that positions evaluation in departments 
produces adequate results; 


* monitoring by the Secretariat of the 
Standard’s application in departments, and 
solid testing based on valid data to 
demonstrate persuasively the degree to 
which the Standard’s objectives are 
achieved; 


+ ability of managers to demonstrate to 
employees that the Standard has been 
applied consistently in their department 
and across departments and for all 
occupational groups; 
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¢ determination of acceptable pay 
levels that enable departments to recruit 
and retain competent people, and 
minimization of the number of positions 
with salary protection upon conversion. 
This is needed to hold organizational 
disruption to a manageable level; and 


* leadership by the Secretariat with a 
clear communications plan and strategy 
for managing change. This will help to 
enlist departments and unions in the 
critical buy-in of the new system and 
ensure that the impact on individuals and 
the organization is well understood. 


22.80 We believe that the effective and 
efficient completion of the Universal 
Classification Standard project requires 
that reasonable and realistic financial 
resources be approved and allocated. We 
expect the Secretariat and departments to 
report the full cost of this project. 


22.81 The Treasury Board Secretariat 
should continue to verify that the 
departments and agencies are 
implementing the Standard 
appropriately, that work descriptions 
are accurate and complete, and that 
evaluation results are valid and 
defensible. Departments should 
continue with diligence and on a 
reasonable timetable to evaluate or 
revise all of their job descriptions. 


22.82 The Treasury Board Secretariat 
should perform all necessary analyses 
based on a valid sample and on 
accurate, up-to-date data to establish 
whether the Standard has reasonably 
achieved its objectives. If it has not, the 
Secretariat should take the necessary 
corrective measures. 


Treasury Board Secretariat’s response: 
The Treasury Board Secretariat agrees 
with and accepts the recommendations in 
the chapter. Monitoring of departmental 
application results will continue. We are 
confident that our increasingly robust 
qualitative and quantitative testing 
methodologies will continue to reinforce 
confidence in the Universal Classification 


Among the key 
conditions for the 
Standard’s success 
are determining 
acceptable pay levels 
that enable 
departments to recruit 
and retain competent 
people and minimizing 
the number of 
positions with salary 
protection upon 
conversion to the 
Standard. 
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Standard as a job evaluation plan that 
measures all work in the public service 
and appropriately respects the Canadian 
Human Rights Act. This testing will also 
continue after implementation to ensure 
that the Standard continues to meet the 
needs of the public service as they change 
over time. 


Conclusion 


22.83 The development of a new 
Universal Classification Standard has 
made a great deal of progress since 

May 1996. There is no doubt that as a job 
evaluation tool, the Standard is an 
enormous improvement over the existing 
classification system. The process the 


Treasury Board Secretariat has followed is 
significantly more rigorous than we 
observed in our 1996 audit. 


22.84 The departments we visited 
strongly support the Standard. However, 
they recognize that they need to do more 
work to ensure that results are of good 
quality before they fully implement the 
new system. 


22.85 In our opinion, the Secretariat 
needs to conduct further rigorous testing 
to demonstrate to stakeholder groups the 
extent to which the Standard has achieved 
its objectives. It needs to continue 
conducting thorough and systematic 
analyses, based on valid and reliable data. 
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Ag foot the Follow-up 


Objectives 


The objective of our chapter was to assess the extent to which the Treasury Board Secretariat has 
implemented our 1996 recommendation (Chapter 5, May 1996 Report) that addressed the development and 
validation of the Standard. 


Our other 1996 recommendations for simplifying the job evaluation system and the occupational group 
structure were not covered by our follow-up. 


Scope 


We reviewed the important documentation used in developing and validating the design of the Standard and 
participated as observers in committees set up by the Secretariat at various stages of the project. 


Between March and June 2000, we studied how eight departments had applied the Standard. We selected 
eight departments that differed in mission, size and composition. They represented about 40 percent of jobs in 
the public service and just over 80 percent of occupational groups. We held structured interviews and 
reviewed progress reports supplied by departments to the Secretariat, as well as other relevant documentation. 


We also participated in discussions of technical and operating issues with such stakeholders as the Canadian 
Human Rights Commission (CHRC), the unions, and pay equity experts. 


Finally, in September 2000 we used a short questionnaire to update our observations on the eight departments 
we visited. 


Criteria 


The criteria used in this audit took into account the government’s objectives for the Universal Classification 
Standard and were drawn from the best practices in the field of job evaluation: 


¢ the method used for collecting information on job content in public service jobs is valid, reliable, credible 
and free of bias, with appropriate quality control mechanisms to permit the collection of information on 
all significant job content, including work associated with women’s work that is often overlooked; 


¢ the sample of jobs used to gather information on job content is adequate and credible, with adequate 
gender representation; 


* the various components of the Standard are reviewed to ensure that they can effectively capture and 
properly value all the significant job content found in the public service, including that associated with 
women’s work. This should include ensuring that: 


- all components of the Standard are clearly defined and measure unique aspects of work to the extent 
possible; 


- the progression from minimum to maximum value in each element is clear; and 
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- illustrations intended to assist raters provide a reasonable spectrum of job content, do not undervalue 
work performed by women and are consistent with the definitions and the desired progression in value; 


° the weight allocated to each element is appropriately distributed to permit the valuation of all jobs, 
including work performed by women, on a fair and equitable basis while giving proper recognition to the 
objectives and values adopted for the public service and the reform; and 


° thorough testing, on a valid sample, of the validity, reliability and gender neutrality of the Standard is 
carried out with adequate quality control measures and, if necessary, review and additional testing before 
implementation. 


Audit Team 


Assistant Auditor General: Maria Barrados 
Principal: Kathryn Elliott 
Director: Ghislaine Cété 


Catherine Gendron 
Marie-Eve Viau 


For information, please contact Kathryn Elliott. 
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Chapter 23 


Information Technology 


Acquisition of Goods and Services 


The audit work reported in this chapter was conducted in accordance with the legislative 


mandate, policies and practices of the Office of the Auditor General. These policies and 
practices embrace the standards recommended by the Canadian Institute of Chartered 


Accountants. The numbered paragraphs in bold face represent recommendations. 
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Standing offer arrangements for commercial software licences and solutions were 
discontinued as a result of bid challenges 

The procurement instrument for replacing software standing offers experienced 
delays and did not provide an adequate solution 
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Information Technology 


Acquisition of Goods and Services 


Main Points 


23.1 In auditing acquisition activities (procurement) for large information technology projects in the 
government, we found that two recent acquisitions followed some essential elements of a procurement approach 
and project management framework designed to improve large technology projects. The approach emphasizes the 
benefits to be derived from a project, rather than focussing on a specific solution to a business problem. Although 
both projects are still in the construction phase, they have progressed and delivered as planned. While full 
implementation and deployment of the new systems are not assured, the results to date are encouraging. 


23.2 We found, however, that the acquisition activities still spanned 18 months or more. The government 
needs to improve the timeliness of those activities to adequately support its objective of putting government 
on-line by 2004. 


23.3 We also examined the acquisition process for the routine supply of microcomputers, equipment for local 
area networks and software. We found that the government has yet to finish replacing its discontinued method of 
supply for routine, low-volume purchases of software licences and software solution services. As a result, user 
departments and agencies do not have a ready, prenegotiated means for low-value purchases of certain software 
products and services. 


23.4 We also found that the replacement method as developed thus far is inadequate, and there is still a risk 
that the government will continue to acquire installed bases of software on a piecemeal and non-competitive basis. 
Where user departments and agencies have done so, the government can become vulnerable to the filing of 
complaints with the Canadian International Trade Tribunal in relation to contracts that extend the use of the 
software to the entire department or involve department-wide maintenance and upgrade of the software. 


23.5 Acquiring software licences and software solution services appropriately continues to be a challenge. The 
government needs to consider a strategy for acquiring software as a technology investment. 


Background and other observations 


23.6 Our data analysis showed that the significant expenditures involved in the acquisition of information 
technology (IT) goods and services have been on the rise. We estimated that in 1998-99, spending on IT 
amounted to about $3.4 billion, including salary costs. 


23.7 The method of supply for routine, low-volume purchases of microcomputers and products related to local 
area networks has merit and adds value to the acquisition process. Nevertheless, in our view there are 
opportunities for cost savings of about $10 million a year by taking more advantage of consolidated purchases. 


23.8 Our audit looked at eight user departments and agencies. We conducted a test of their compliance with 
the procurement authority delegated to them. We found a few minor deviations but no indication of blatant or 
widespread misuse of delegated authority. 


23.9 Recent determinations by the Canadian International Trade Tribunal (CITT) were the main reason for 
changing the standing offer method of supply for routine, low-volume purchases of software licences and software 
solution services. Moreover, even though CITT cases have been few, government users and the IT community 
have indicated that they have generated intense scrutiny of all technology acquisitions in government in order to 
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avoid potential complaints to the Tribunal. In our view, there is a need to better respect the government’s 
contracting objectives — meeting operational requirements with due regard to value for money while also 
respecting the principle of open and fair competition by complying with government regulations and trade 
agreements. 


The government is generally supportive of the recommendations contained in the chapter. 
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Introduction 


Acquisitions are made in a highly 
complex environment 


23.10 The acquisition or procurement 
of information technology (IT) goods and 
services in the government is a large and 
complex business. In recent years, the 
government has made major investments 
in information systems and technology to 
support the delivery of programs and 
services to Canadians. Its acquisitions 
range from small volumes of low-cost 
goods to large purchases for multi-year 
development projects that integrate IT 
goods and services from different 
suppliers. 


23.11 Unlike the private sector, the 
government must follow many rules and 
regulations in its acquisitions, including 
its purchases of IT goods and services. 
Principal among them are the contracting 
policy, the Treasury Board Contracts 
Directives, the Government Contracts 
Regulations (pursuant to the Financial 
Administration Act) and the Supply 
Manual. 


23.12 The Treasury Board Secretariat is 
responsible for the government’s 
contracting policy. Public Works and 
Government Services Canada (PWGSC) 
acts as the contracting authority for most 
government acquisitions and is in charge 
of the Supply Manual — the procedural 
manual for its supply activities. 
Government acquisitions are also subject 
to multilateral and federal-provincial trade 
agreements: the North American Free 
Trade Agreement, the World Trade 
Organization—A greement on Government 
Procurement, and the Agreement on 
Internal Trade. The Canadian 
International Trade Tribunal, an 
administrative tribunal, conducts inquiries 
into complaints by potential suppliers 
about government procurement actions, 
where they are covered by the trade 
agreements. 
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23.13 The contracting policy states: 


The objective of government 
procurement contracting is to acquire 
goods and services and to carry out 
construction in a manner that 
enhances access, competition and 
fairness and results in best value or, if 
appropriate, the optimal balance of 
overall benefits to the Crown and the 
Canadian people. 


Policy statements further underscore the 
pre-eminence of operational requirements; 
the ability to withstand the test of public 
scrutiny; support for industrial and 
regional development and other 
appropriate national objectives; and 
compliance with the government’s 
obligations under trade agreements. 


23.14 At the user level, buying IT 
goods and services can involve several 
groups of staff in a government 
department. The acquisition process 
generally involves the users — program 
managers with operational needs to be met 
— and those IT staff who have knowledge 
of and expertise in technology matters as 
well as those who may be affected directly 
by the acquisition of a particular 
technological solution. It may also involve 
corporate services staff engaged in 
materiel management or representing the 
department’s contracting authority. In 
departments where operations are highly 
decentralized, additional groups may be 
involved. 


Users sometimes find 
themselves with 
problems whose 
solutions are difficult 
to anticipate or fully 
understand at the time 
of procurement. 


23.15 The government’s procurement 
environment, already complex and 
involving many stakeholders, is 
complicated further in the field of 
information technology. Information 
technology evolves at a very rapid pace. 
New products and technological solutions 
are constantly being developed. Users 
sometimes find themselves with problems 
whose solutions are difficult to anticipate 
or fully understand at the time of 
procurement. Moreover, current 
acquisitions often involve major 
investments and can have significant 
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implications for choices of technology in 
the future. 


23.16 In this complex environment, the 
government is challenged to meet 
operational needs with a solution that 
provides best value while enhancing 
access, competition and fairness in its IT 
procurement activities. 


Focus of the audit 


23.17 The objective of our audit was to 
assess the extent to which IT acquisitions 
in the federal government meet policy 
objectives while giving due regard to 
efficiency and economy. 


23.18 The audit focussed on the 
acquisition process and activities for large 
IT projects and for the routine supply of 
microcomputers, equipment for local area 
networks, and software. As it is discussed 
in this chapter, the process starts when 
users identify an operational need and 
ends when a contract has been awarded 
and any challenges to the award have been 
resolved. 


23.19 We analyzed data on IT 
expenditures to understand the extent and 
the nature of IT acquisitions in 
government. We reviewed the processes, 
methodology and tools that have been 
established to facilitate IT procurement 
activities, and we looked at procurement 
cases that demonstrated their use. 


23.20 We conducted interviews at the 
Treasury Board Secretariat, PWGSC, the 
Canadian International Trade Tribunal and 
eight departments and agencies, where we 
also examined cases of IT acquisitions to 
support and illustrate potential issues. We 
examined several IT acquisitions that had 
resulted in challenges before the Canadian 
International Trade Tribunal. 


23.21 Further information about the 
audit objective, scope and criteria can be 
found at the end of the chapter in About 
the Audit. 


Observations and 
Recommendations 


Analyzing Information Technology 
Expenditures in Government 


Government-wide data are not available 


23.22 To better understand the 
acquisitions of IT goods and services, we 
asked for information on IT expenditures 
incurred by the government. From our 
discussions with the Treasury Board 
Secretariat and Public Works and 
Government Services Canada, we learned 
that data are not specifically coded as IT 
expenditures and thus are not aggregated. 
While the Secretariat annually reports on 
total contracting in government, no 
reporting on government-wide IT 
acquisitions is available. (The subject of 
contract reporting is addressed in Chapter 
29 of this Report.) 


23.23 Inthe absence of such 
information, we set out to analyze IT 
expenditures government-wide and by 
department as well as by type of goods 
and services acquired. Our analysis also 
included a general estimation of payroll 
costs in support of IT development and 
operations. The analysis spanned four 
fiscal years, from 1995-96 to 1998-99. 


23.24 Based on expenditure data from 
the Public Accounts of Canada, we 
estimated a rough order of magnitude for 
IT-related expenditures other than payroll 
by using selective codes of economic 
object. To estimate IT-related payroll 
expenditures, we used portions of salary 
and employee benefit costs, by pay 
classification, from data provided by the 
Public Service Commission. We also 
sought input from the Treasury Board 
Secretariat in making our estimates. 


23.25 Our estimation was conservative 
in nature. For example, we did not attempt 
to include expenditures coded for 
specialized work such as constructing 
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satellites or other highly sophisticated 
equipment, like weaponry systems. 


23.26 Our analysis confirmed that IT 
expenditures are significant and rising. 
Exhibit 23.1 shows that IT expenditures 
increased from almost $3 billion in 
1995-96 to about $3.4 billion in 1998-99, 
or by almost 14 percent. During the same 
period, spending on IT as a proportion of 
total budgetary expenditures for 
government operations increased from 
8.6 percent to 10.1 percent. 


23.27 Our analysis also showed that 
four departments, each with annual IT 
spending of more than $100 million, 
accounted for about 60 percent of the total 
expenditures on IT in the government. 
Departments and agencies with moderate 
IT expenditures (from $25 million to 
$100 million a year) accounted for almost 
30 percent of the total. Exhibit 23.2 shows 
IT expenditures of departments and 
agencies by category of large, moderate or 
small amount spent in the past four years. 
In summary, about 20 departments with 
the highest spending on IT accounted for 
roughly 90 percent of the government’s 
total IT expenditures. 


23.28 We also set out to analyze IT 
expenditures by type of purchase, such as 
hardware, software, professional services, 
and telecommunication products and 
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services. However, because departments 
and agencies were inconsistent in the way 
they applied economic object codes and 
coded their expenditures for large IT 
projects, it was not feasible to complete 
that analysis in a meaningful way. 


Procuring Large Information 
Technology Projects 


23.29 Large projects to develop 
information systems as business solutions 
often meet with failure to varying degrees, 
from not meeting business needs fully to 
not delivering and implementing the 
systems on time, to incurring excessive 
costs. This general observation was 
supported by our audits of government 
systems under development that we 
reported to Parliament in 1995, 1996 and 
1997" 


23.30 Past research has shown that 
there are a number of major factors that 
cause projects to fail. The acquisition 
phase and the related procurement 
activities are one such factor. At that time, 
the process of procurement for large IT 
projects typically resulted in multi-year, 
fixed-price contracting. The contracts 
would include detailed specifications and 
requirements, and the lowest bidder, based 
on a combined evaluation of technical 
merits and price, would be awarded the 
contract. 


Information 
technology 
expenditures are 


significant and rising. 


($ millions) Percent Exhibit 23.1 
3,600 _ = 
eee 20 Information Technology Expenditures 
3.000 Are Significant and Rising 
Seo 
2,400 — 
49 Payroll costs [| 
1,800 — 
— 9 Non-payroll costs [J] 
8 pay 
1200. — 
IT expenditures as a percentage *—* 
600 — & 4 of total budgetary expenditures 
for government operations 
0 — ram a 
1995-96 1996-97 1997-98 1998-99 
Source: Receiver General and Public Service Commission 
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Exhibit 23.2 


Spending on Information 
Technology by Departments and 
Agencies 


WB $100,000,000 or more 


|_| between $25,000,000 and $100,000,000 


[_] below $25,000,000 


Source: Receiver General 


New framework and approach 
developed to improve results 


23.31 In 1996, PWGSC undertook to 
develop a new procurement methodology 
that would address some of the 
shortcomings of the existing approach to 
large and complex IT acquisitions. The 
Department collaborated with the 
Treasury Board Secretariat, 
representatives of user departments and 
agencies, and industry associations 
representing suppliers. In March 1997, 
PWGSC released the second version of 
the Benefits Driven Procurement (BDP) 
approach as a solution for the acquisition 
phase of large IT projects. 


23.32 The BDP approach emphasizes 
the results and benefits to be derived from 
a project, rather than emphasizing a 
specific solution to a business problem. 
Thus, it focusses on making a strong 
business case. Instead of the detailed 
specifications and requirements of a 
solution, the approach highlights the 
benefits that the solution will provide. The 


project is delivered in phases that provide 
“off-ramps” — opportunities to halt the 
project, if appropriate. Compensation is 
based on performance, usually at the 
completion of defined tasks. In addition, 
BDP supports clear accountability by all 
parties to the contract, and sound project 
management disciplines that include 
rigorous identification and management of 
tisk. Since its development, the approach 
has been recommended for use in 
procurements for large and complex IT 
projects. 


23.33. The BDP approach is fully 
consistent with the Enhanced 
Management Framework (EMF) for 
managing large IT development projects. 
The Treasury Board Secretariat developed 
the EMF in 1995 to address weaknesses 
that we had identified in our audits of 
systems under development. The elements 
of BDP, such as the requirement for a 
strong business case with periodic updates 
and the emphasis on up-front planning, are 
also key elements of the Framework. 


Number of Departments and Agencies 
(percentage of total IT expenditures) 


4 (62%) 


13 (28%) 
1995-96 


4 (63%) 


63 (10%) 


14 (27%) 
1997-98 


OF 500% 


4 (62%) 


61 (8%) 


15 (30%) 


1996-97 


4 (62%) 


Pe / (8%) 


15 (30%) 
1998-99 
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Few projects are available for review 
but new concepts are being used 


23.34 With the development of the 
EMF and the BDP approach, we would 
expect current acquisitions for large IT 
projects to apply their essential elements. 


23.35 We set out to identify large IT 
acquisitions made in the past few years in 
order to select a sample for review. 
However, meetings with officials of the 
Treasury Board Secretariat, PWGSC and 
the selected user departments revealed 
that except for two major initiatives, few 
large IT projects have been launched in 
recent years. 


23.36 The Year 2000 computer problem 
was a top priority of the government for 
most of 1998 and 1999. Most systems 
development work was deferred in favour 
of remediation and testing of 
non-compliant systems and devices, 
especially those considered 
mission-critical. As a prudent measure, 
most departments, even after putting 
compliant systems in place, resisted 
making changes in their systems to avoid 
any risk that their computer environment 
would be non-compliant for Year 2000 as 
a result. At the same time, departments 
and agencies were also converting their 
financial systems or implementing new 
systems to meet the requirements of the 
government’s Financial Information 
Strategy by its 2001 deadline. We have 
audited the Strategy in the past and will be 
conducting more audits of it in the future. 


23.37 We focussed this audit on large 
IT projects that are using the concepts and 
elements of the newly developed approach 
and framework. Aside from IT work 
related to Year 2000 and the Financial 
Information Strategy, very few large IT 
projects have been undertaken in the last 
two years. Nevertheless, we identified two 
recent large IT acquisitions — Export and 
Import Controls System and Materiel 
Acquisition and Support Information 
System — and reviewed their acquisition 
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phase. The Year 2000 omnibus contract 
has been cited as an example of another 
procurement strategy. We conducted a 
limited review of that procurement. 


23.38 The Export and Import Controls 
System (EICS) project at Foreign Affairs 
and International Trade was designed to 
support the Department’s control over 
export and import activities under the 
Export and Import Permits Act. It was 
intended to replace two existing systems 
that were becoming obsolete, and to 
provide interfaces with key stakeholders 
in industry and in other government 
entities. 


23.39 National Defence had earlier 
made a number of attempts to optimize 
the availability of its equipment and 
weaponry systems. The Materiel 
Acquisition and Support Information 
System (MASIS) project in its current 
form is aimed at optimizing their 
availability throughout their life cycle. It 
is meant to provide an integrated 
environment for Canada’s Air, Land and 
Marine Forces and the Department’s 
administrative units, to replace the 
separate information systems that have 
managed and supported the military 
equipment and weaponry systems of each 
Force. 


23.40 The “Year 2000 omnibus 
contract” was an initiative to ensure the 
availability of skilled technical resources 
for Year 2000 remediation, testing and 
implementation work. Under the 
initiative, seven contracts were awarded 
and they formed a supply arrangement for 
departments and agencies. The 
arrangement guaranteed an overall 
minimum business volume to the 
suppliers. The initiative initially applied 
only to work on systems that support 
government-wide mission-critical 
functions. It was later extended to cover 
systems that support departmental 
mission-critical functions. The supply 
arrangement was established for work on 
all mission-critical systems. 


LT 
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The business case is a 
cornerstone of 
successful 
implementation of 
large, complex IT 
projects. 


23.41 The two IT projects are in 
various stages of implementation. The 
Year 2000 contracts expired in June 2000. 
Exhibit 23.3 summarizes the acquisition 
phase for the two projects and provides 
selected information on the Year 2000 
contracts. 


23.42 Developing a business case. The 
business case is a cornerstone of 
successful implementation of large, 
complex IT projects. Developing a strong 
business case is at the core of the Benefits 
Driven Procurement (BDP) approach and 
is a key element of the Enhanced 
Management Framework (EMF). Both IT 
projects we looked at developed 
comprehensive business cases. 


23.43 The business case for the Export 
and Import Controls System (EICS) 
project identified the desired outcomes 
and benefits that the project was to 
deliver. It related those benefits to the 
mandate of Foreign Affairs and 
International Trade and to the program 
direction of the Export and Import 
Controls Bureau. The business case also 
contained an analysis of several potential 
solutions to the business problem, 
including their long-term costing 
implications. The rationale for the 
recommended solution was also 
documented. 


23.44 The Materiel Acquisition and 
Support Information System (MASIS) 
project was supported by a business case 
that described the re-engineering 
challenge and the benefits of integrating 
information systems maintained at the 
time by separate Forces and 
administrative units. The business case 
has been reviewed as the project has 
developed, and adjustments have been 
made as appropriate. All “off-ramp” 
decisions, new tasks and changes in scope 
are governed by the business case. It is 
also the basis for assessing performance. 


23.45 Using a phased approach. By 
breaking a project down into phases, the 
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BDP approach provides checkpoints for 
deciding whether the benefits accruing to 
end users can no longer justify a further 
investment of resources. Both projects 
used a phased approach with successive 
off-ramps. In addition, both departments 
used the first phase of the project to 
develop the strategy for the business 
solution and to define the development 
and construction phase of the project. 
Both projects emphasized the overall 
benefits to be achieved, rather than 
defining detailed specifications and 
requirements at the outset. 


23.46 For the EICS project, the 
Department contracted with a consulting 
firm to help prepare the request for 
proposals, including evaluation criteria. 
The project itself involved a definition 
phase and a construction phase. The 
contract for the definition phase ended in 
January 2000 with the successful 
completion of formal system 
requirements. The contract for the 
construction phase contains many tasks 
that are used as checkpoints, providing 
additional opportunities for either party to 
exit the contract if desired. Full 
deployment of the project is expected in 
April 2001. 


23.47 For the MASIS project, National 
Defence began the formal acquisition 
process by releasing a draft request for 
proposal. The final request for proposal 
contained minimal functional 
specifications and it was expected that the 
IT industry would provide the solution. 
The contract for the first phase involved 
not only developing an overall system 
design and implementation plan but also 
replacing an obsolete system at the 
Department’s workshop depot in Montreal 
with at least 30 percent of the 
functionality planned for the new system. 
The contract for the second phase deals 
with the construction and delivery of the 
full system. Deployment of the overall 
system involves incremental rollouts with 
off-ramps, and is expected to be 
completed by June 2004. 
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Exhibit 23.3 


Acquisition Phase of Two Large Information Technology Projects and the Year 2000 Supply Arrangement - Summary 


Materiel Acquisition and Export/Import Year 2000 Supply 
Support Information System Controls System Arrangement 
¢ Integrated information system | * Control system to support the | * Professional services contracts | 
designed to enable the Navy, control aspects of trade made available to departments 
Army and Air Force to agreements, quota allocation for the provision of Year 2000 
maintain and support their policy and ministerial permit remedial solutions for 
equipment and weapon issuance mission-critical systems 
systems more effectively * Interfaces with exporters, * Set up with 7 firms to speed 
¢ Based on commercial software importers, Customs brokers, up and facilitate the provision 
and industry’s best business and Canada Customs and of needed IT professional 
practices in materiel Revenue Agency services 
acquisition and support * Expected cost $6 million * Budgeted for a minimum 
gi a $100 million 
* Expected cost $109.3 million 
[ Milestone Activity 
is 
| Recommendation to develop a October 24, 1994 
joint solution for all the Forces 
Request for proposal (for March 1996 
| pilot project) (bids later rejected) 
Further consultations with July to December 1996 
industry and other countries 
| Decision to use Benefits January 19, 1997 Fall 1997 Fall 1997 
| Driven Procurement approach 
Letter of interest or draft request | March 1998 December 4, 1998 October 3, 1997 | 
for proposal, posted on MERX 
Request for proposal June 26, 1998 January 27, 1999 November 7, 1997 
Closing date August 4, 1998 March 17, 1999 January 7, 1998 
Supply arrangement contracts Early March 1998 
awarded to 7 firms 
First call-ups June 1998 
Completion June 30, 2000 
Ist phase contract awarded: 
Initial solution December 11, 1998 
Definition phase May 25, 1999 
Completion June 30, 2000 January 18, 2000 
2nd phase contract awarded: 
Full functionality solution June 30, 2000 | 
Construction phase [ December 24, 1999 7 = = _| 
(Planned) 
Incremental rollouts with until April 1, 2004 
off-ramps 
Final deployment April 1, 2001 7 : y 


Source: Department of Foreign Affairs and International Trade; National Defence; Public Works 
and Government Services Canada. 
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23.48 Establishing governance and 
getting senior-level sponsorship. Senior 
sponsorship and appropriate governance 
are key ingredients in successfully 
implementing large IT projects. We found 
aspects of both in the two projects that we 
reviewed. Moreover, there has been no 
turnover among the several senior officials 
who are responsible for the projects. 


23.49 The Director General of the 
Export and Import Controls Bureau has 
served as the senior sponsor for the EICS 
project since the beginning. The project is 
guided by a steering committee, 
comprising senior departmental officials 
and a senior representative of the 
contractor firm. The respective roles and 
responsibilities of the Department and the 
contractor are stated in each of the two 
contracts. 


23.50 For MASIS, the sponsor is a 
senior official of National Defence’s 
Materiel Comptrollership and Business 
Management Directorate. The project 
manager is a senior officer of the Materiel 
Group and has been in that capacity since 
1996. Internally, the project team reports 
to the MASIS Senior Review Board, 
which consists of senior staff from all 
three Forces, Defence Information 
Services, Logistics, and Engineering. An 
Executive MASIS Steering Committee 
oversees the project. The Steering 
Committee includes senior officials of all 
stakeholders — National Defence, 
PWGSC and the contractor firm. 


23.51 Managing the project and 
related risks. Although it has not been 
identified as a key element of the BDP 
approach and the Enhanced Management 
Framework, an integrated team has been 
used to manage each of the two projects 
we reviewed. Integrated project 
management teams comprise staff of the 
department and consultants from the 
contractor firm. Use of the teams provides 
for flexibility and speed in making 
changes to advance the project or to 
mitigate risks. 


23.52 The Integrated Project 
Management Office for EICS involves 
departmental officials with participants 
from the contractor firm. It is chaired by 
the project authority from the Export and 
Import Controls Bureau, and a senior 
official in the Department’s Information 
Management and Technology Bureau is 
also a member. Each month, the Office 
updates a detailed risk analysis of the 
project and reviews the updated analysis 
to determine whether changes or 
additional actions are warranted. The 
contractor firm receives payment upon the 
successful completion of authorized tasks 
by the target dates. The contract also 
contains provisions for avoiding or 
resolving disputes. 


23.53 Inthe case of MASIS, the 
requirement to use an integrated project 
team was stipulated in the Statement of 
Work for each phase of the project. In 
addition to providing for flexibility and 
quick action, the integrated team approach 
is expected to increase efficiency, improve 
communication and streamline 
management oversight. Like EICS, the 
MASIS request for proposal stated that a 
risk management plan was a requirement. 
Each phase of the project was broken 
down into a series of subprojects with 
defined tasks. Payments are subject to the 
successful completion of pre-authorized 
tasks. In addition to remedy clauses, the 
MASIS contracts provide for incentive 
conditions to be negotiated for superior 
performance. 


Outcomes are not definitive but results 
to date are encouraging 


23.54 Both EICS and MASIS are 
currently in the construction phase. 
Although full implementation and 
deployment are not assured, both projects 
have progressed and delivered as planned. 
At July 2000, they had met expectations 
on schedule and stayed within budgeted 
costs. 


23.55 The first-phase contracts for the 
EICS and MASIS projects were 
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successfully completed and formed the 
basis for the work under the second-phase 
contracts. The deployment of EICS is 
expected in April 2001, as planned. For 
MASIS, by September 1999 the 
first-phase contract had provided systems 
capability for 30 percent of functional 
requirements at the Department’s 
workshop depot in Montreal. With 
incremental rollouts, full deployment of 
the project is expected by June 2004. 


23.56 The nature of the Year 2000 
contracts differed from that of the two 
systems development projects. Its purpose 
was to ensure the availability of skilled 
technical resources. It did so using a 
supply arrangement that featured a 
guaranteed minimum of $100 million in 
contract work among the seven successful 
bidders. Through the supply arrangement, 
a user department could select directly 
from the seven prequalified contractor 
firms and have PWGSC solicit work plans 
from them. By June 2000, when it ended, 
the supply arrangement had met 
expectations in providing skilled contract 
resources. The full guarantee had been 
exercised and there had been virtually no 
system failures. We also noted an 
important benefit — it provided a 
streamlined procurement process for 
contracting technical resources and 
allowed for their rapid deployment. The 
application of this supply arrangement 
was unique. Yet it serves as an example 
that could be adapted to benefit future 
procurement activities. 


23.57 Our examination of the two large 
IT projects showed that many essential 
elements of the Benefits Driven 
Procurement approach and the Enhanced 
Management Framework can work and 
produce results. Analyzing experience 
from two projects is not sufficiently 
conclusive, and both projects have yet to 
be completed to demonstrate full success. 
However, experience in both EICS and 
MASIS to date has been encouraging. 
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23.58 In our view, the emphasis on 
front-end planning was an essential 
element in the positive outcome of the two 
projects to date. Its merit goes beyond 
serving as guidance and recommended 
procedure. It ought to be required as a 
general practice in procuring and 
developing large IT projects. The 
front-end planning included: 


¢ focussing on business needs, defined 
in terms of program mandate and business 
outcomes, and not on the solution; 


* breaking the project down into 
phases and tasks; 


* obtaining the commitment and 
ongoing involvement of senior 


management; In our view, the 


emphasis on front-end 
planning was an 
essential element in 
the positive outcome 
of the two projects to 
date. 


¢ establishing a governance structure; 
and 


* involving PWGSC early in the 
development of an appropriate 
procurement strategy. 


23.59 The Treasury Board Secretariat 
and Public Works and Government 
Services Canada should analyze the 
experience of recent acquisitions and 
specify the practices that they will 
generally require of user departments 
and agencies in large information 
technology projects. 


Government’s response: Treasury Board 
Secretariat and Public Works and 
Government Services Canada support the 
recommendation. We will continue to 
support the application of appropriate 
Enhanced Management Framework and 
Benefits Driven Procurement practices to 
large information technology projects. 


Challenges remain in future 
procurement for large projects 


23.60 We have noted some positive 
aspects of using the new procurement 
approach and the project management 
framework. However, there are issues that 
still need to be addressed. 


23.61 Technology advances at a rapid 
pace. New products with ever-increasing 
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The longer it takes to 
acquire, develop and 
implement 
technological 
solutions, the higher is 
the risk that the 
business needs will 
change and the 
solutions become less 
beneficial and 
relevant. 


processing speed and capacity are 
constantly evolving. Moreover, the 
workplace changes continually, along with 
program requirements and operational 
needs. The longer it takes to acquire, 
develop and implement technological 
solutions, the higher is the risk that the 
business needs will change and the 
solutions become less beneficial and 
relevant. Long phases of acquisition and 
implementation also increase the risk that 
the acquired technology will become 
obsolete by the time it is deployed. 
Consequently, acquisition activities that 
require shorter time spans will add 
significant value to projects. 


23.62 We reviewed the chronology of 
activities of the two large projects and 
analyzed the time span of the acquisition 
activities. We found that they spanned 
periods exceeding 18 months. The EICS 
project needs were first analyzed in 1997 
and the first phase contract was awarded 
in late May 1999. Prior to adopting the 
BDP approach, the need for MASIS had 
been identified and attempts had been 
made to develop it as early as 1994. The 
MASIS project in its current form was 
initiated in 1996 and its first-phase 
contract was awarded in December 1998. 


23.63 The IT industry claims that 
processing speed doubles in 18 months, 
and it defines a Web year as three 
calendar months. Development and 
deployment of large IT projects are not 
expected to match this pace of change. 
However, the government still needs to 
find ways to reduce the time span of its 
acquisition activities in large IT projects. 


23.64 In reviewing the procurement 
strategy for the Year 2000 omnibus 
contract, we noted that it provided an 
instrument for rapid deployment of 
contract resources, and its experience can 
benefit future procurement. But there are 
related factors that need to be considered 
when replicating this method of supply. 


23.65 Year 2000 remediation was a 
unique project. The goal was to ensure 
that critical functions and operations 
would continue upon the arrival of 2000 
and beyond. There was a degree of 
commonality in the resource requirements 
of user departments and agencies. That 
commonality facilitated the structuring of 
the supply arrangement. In large IT 
projects, there is no expectation of a 
natural tendency toward commonality. 
The challenge remains to define 
categories of technical resources for 
purposes of establishing a supply 
arrangement. 


23.66 Another unique factor in the 
Year 2000 contracts was the cost. Since 
the purpose of the contract was to assure 
availability, successful bidders had to 
guarantee that they would make technical 
resources available if and when called 
upon. To retain staff who were in 
simultaneous demand worldwide, the rates 
of compensation and incentives had to 
reflect the North American marketplace 
for Year 2000 work. Consequently, 
contract rates were set at a premium and 
contractor firms were guaranteed a 
minimum of $100 million in work. As the 
government extends this procurement 
strategy to other acquisitions, rate 
negotiation will become an important 
factor in controlling costs and providing 
value for money. 


23.67 Recent experience with 
acquisitions for large IT projects has been 
promising. However, the time entailed in 
the acquisition phase could become an 
impediment to the successful 
implementation of technology solutions 
that meet business requirements. The 
government needs to build upon the BDP 
approach to improve its acquisition 
activities, using the procurement process 
to advance and support large IT 
acquisitions. 


23.68 The government should work 
with the information technology 
industry to further develop its 
procurement approach for large, 
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complex technology projects in a way 
that will improve its acquisition 
activities, particularly their timeliness. 


Government’s response: We support the 
recommendation and agree that we should 
work with industry to facilitate timely 
procurement for large and complex 
projects. We are currently doing so in 
pursuit of an optimal procurement strategy 
for the Government On-Line (GOL) 
requirements. 


Acquiring Software, 
Microcomputers and Equipment 
for Local Area Networks 


23.69 Standing offers are a procurement 
instrument used by PWGSC that allows 
user departments and agencies to order 
directly from suppliers. They provide a 
means to acquire goods and services at 
prenegotiated prices that remain in effect 
for a specified period of time. Standing 
offers are the preferred method for 
routine, low-volume supply of commercial 
off-the-shelf software, microcomputers 
and equipment for local area networks 
(LANs). 


23.70 Within the limits of delegated 
authority, user departments and agencies 
can issue call-ups against a standing offer 
with a specific supplier, requesting the 
exact quantities of goods and services they 
require. Generally, when a call-up is made 
against a standing offer, there is no further 
negotiation of price and the supplier is 
obliged to meet the call-up requirement. 
To allow for the rapidly changing 
environment, a supplier is permitted to 
substitute listed products with improved or 
upgraded versions, provided they meet or 
exceed benchmarking requirements and 
are offered at the same or lower prices. 


23.71 There are several different types 
of standing offer. The national master 
standing offer (NMSO) is the most 
common. The NMSOs are arranged by 
PWGSC to meet common needs of 
departments and agencies and to provide 
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them with routine supplies. The NMSOs 
are arranged without requisitions from 
users. Regional standing offers are also 
used to satisfy needs within a specific 
geographic area. Furthermore, there are 
individual standing offers used to meet the 
needs of a single department or agency. 


Method of supply of microcomputers 
and products related to local area 
networks has been established to 
provide value for money 


23.72 Public Works and Government 
Services Canada has arranged national 
master standing offers for routine, 
low-volume purchases of microcomputers 
and LAN-related products, including 
desktop and notebook computers and 
servers. The standing offers are awarded 
competitively to suppliers that meet 
technical and benchmarking requirements. 
In order to qualify, the supplier must also 
offer a product at a price within a defined 
range of the average price submitted by 
other suppliers for products in the same 
category. 


23.73 In general, a user department can 
issue a call-up for desktop and notebook 
computers directly to a specific NMSO 
supplier, up to a maximum of $40,000. 
The limit for servers is $100,000 per 
call-up. Information on existing NMSOs is 
updated regularly on the PWGSC Web 
site, under its Computer Acquisition 
Guide. The site is open to public access to 
encourage competition among suppliers. 


23.74 The NMSOs for desktop 
computers are the most heavily used. On a 
biweekly basis, PWGSC assesses products 
and identifies those that offer the lowest 
cost in each of the desktop categories, 
based on a combined evaluation of 
technical merits and price. The products 
with the lowest evaluated costs are posted 
on the Department’s Web site. To 
encourage the use of those products, 
PWGSC raises their call-up limit from 
$40,000 to $100,000. 


23.75 Standing offers are arranged to 
accommodate routine, low-volume 


Standing offers are 
awarded competitively 
to suppliers that meet 


technical and 
benchmarking 
requirements. 
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Exhibit 23.4 


purchases. To support larger-volume 
purchases, PWGSC provides user 
departments and agencies with an option 
to issue a request for volume discount 
(RVD) to NMSO suppliers on their listed 
prices. The RVDs can be used for 
purchases up to $10 million. They are 
processed on a competitive basis and are 
thus an alternative to the traditional 
competition that involves issuing requests 
for proposals. 


23.76 In our view, the arrangement of 
standing offers as a method of supply for 
microcomputers and LAN-related 
products has merit, and the technical and 
benchmarking requirements add value to 
the acquisition process. 


Opportunities exist for more cost 
savings 


23.77 —Call-ups on an NMSO are 
intended for low-volume purchases. In 
exercising a call-up, a department usually 
purchases the NMSO product at the 
specified price. That price has been set 
within a competitive price range but 
usually at a premium, because of the low 
purchase volume anticipated. Although 


the RVD process was available for 
obtaining volume discounts, most 
microcomputers and LAN-related 
products purchased in the last two years 
were acquired through departments’ 
call-ups. 


23.78 We reviewed information from 
Public Works and Government Services 
Canada on purchases of microcomputers 
and LAN-related products in 1998-99 and 
1999-2000. We noted that on average, 
RVD procurements accounted for around 
25 percent of about $200 million in total 
annual acquisitions of those products. The 
data indicate that call-ups on NMSOs 
were widely used; they accounted for 
about 45 percent of purchases of the 
products. The remaining purchases were 
made using requests for proposals or 
through unchallenged advance contract 
award notices (ACANs). Moreover, about 
65 percent of all microcomputer/LAN 
products that were purchased using 
standing offers were acquired in the last 
three months of the fiscal year. An 
analysis of fourth-quarter data from four 
sample departments confirmed that 
low-volume purchases dominate (see 
Exhibit 23.4). We analyzed a sample of 
RVD transactions processed by PWGSC in 
1999-2000 and found that savings from 
volume discounts averaged about 


Low-Volume Purchases Dominate 17 percent 


Analysis of fourth quarter data from four sample departments 
23.79 Some users expressed the 


(Percentages represent the proportion of purchases that were concern that RVDs may take a long time. 
low-volume) However, in a small sample of RVD 


acquisitions made in 1999-2000, 

rae eer (a s4x) procurement averaged four weeks — a 
ie significant advantage over the traditional 

Department of Fisheries competitive process using requests for 

OSES proposals. Other users who had 

reservations about using RVDs cited their 

preference for retaining sole control or 

their concern that PWGSC might lack 


(67%) 


Human Resources 


Development Canada gy °7%) 


National Archives of Canada []a00%) flexibility 
} | ! ! 23.80 We recognize that RVD 
0 2,000 4,000 6,000 procurement may not always be feasible, 
($ thousands) and there are occasions when simple 
Source: Departmental data on 1999-2000 acquisitions of microcomputers call-ups oe appropriate oh one the 


and local area network products. best value. However, increasing the use of 
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RVD procurement could lead to cost 
savings in the neighbourhood of 
$10 million every year. 


23.81 For expediency, departments and 
agencies use acquisition cards and local 
purchase orders for purchases under 
$5,000. The PWGSC data do not include 
those purchases, and their extent is not 
known. In smaller or highly decentralized 
departments, such purchases can represent 
a significant portion of the department’s 
total procurement. Some of those 
purchases could represent another source 
of cost savings. 


23.82 Added savings or faster 
turnaround times can also be achieved by 
using other forms of standing offer. Three 
of the eight user departments that we 
selected for case review have 
requisitioned their own standing offers. 
National individual standing offers 
(NISOs) are arranged by PWGSC after it 
receives funded requisitions from a 
department. That department can then use 
them throughout Canada. 


23.83 The Department of Foreign 
Affairs and International Trade (DFAIT) 
used NISOs to address its requirement for 
maintenance services at missions around 
the world and to shorten the procurement 
period. In reviewing its data, we found 
that by using NISOs, the Department 
shortened the procurement period for 
certain acquisitions to two weeks. 
National Defence achieved up to 

39 percent savings with its omnibus NISO, 
which expired in July 1999. DFAIT had 
issued six NISOs by August 2000; 
National Defence was experiencing some 
delay in replacing the expired omnibus 
NISO but was continuing its efforts. 


23.84 The national master standing 
offer as a method of supply is well suited 
to acquiring commodities like desktop and 
laptop computers. Nevertheless, we 
believe that additional efforts can be made 
to ensure value for money. 
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23.85 Departments and agencies, in 
co-operation with Public Works and 
Government Services Canada, should 
strive to consolidate purchases of 
microcomputers and LAN-related 
products to take advantage of 
opportunities for additional cost 
savings. Further, Public Works and 
Government Services Canada and user 
departments should consider other 
options to provide enhanced flexibility 
to meet the government’s business 
needs. 


Government’s response: We support the 
recommendation in general. 


Standing offer arrangements for 
commercial software licences and 
solutions were discontinued as a result 
of bid challenges 


23.86 As well as purchasing 
microcomputers and LAN-related 
products, user departments and agencies 
were using NMSOs to purchase 
commercial off-the-shelf software 
products directly from suppliers, subject 
to the authorized limits. Those limits were 
$40,000 for software licences and $50,000 
for software solutions involving services. 
Purchases above the limits were processed 
by PWGSC. In addition to using call-ups, 
departments and agencies made direct 
purchases using acquisition cards, local 
purchase orders and sole-source contracts. 


23.87 The NMSO method of supply 
was established to meet ad hoc needs for 
limited quantities. It was considered an 
efficient procurement method and was 
widely used. During 1998-99, PWGSC 
processed a total of about $200 million in 
software purchases, of which about 

$120 million involved NMSO 
transactions. In mid-1999, there were 65 
NMSOs in effect with software vendors. 


23.88 The NMSOs were awarded to 
software vendors without competition. 
The criteria, unlike those of NMSOs for 
microcomputers and LAN-related 
products, were based solely on previous 


Increasing the use of 
requests for volume 
discount in 
procurement could 
lead to cost savings in 
the neighbourhood of 
$10 million every year. 
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volumes of business and did not involve 
technical specifications or pricing. Using 
this and other non-competitive methods of 
supply, over time departments could build 
up a substantial investment in certain 
software products acquired piecemeal and 
on a non-competitive basis. This could 
become a significant procurement 
problem if a department needed to extend 
the use of the software to the entire 
department, or when it needed to procure 
department-wide maintenance services 
and software upgrades. 


23.89 In recent years, potential bidders 
have successfully challenged the 
government’s solicitation to upgrade 
existing installed bases of software. After 
those cases, PWGSC limited the use of the 
software NMSOs in an attempt to better 
control software acquisitions and guard 
against establishing other installed bases 
of software acquired piecemeal. In August 
1997, the Department and the Treasury 
Board Secretariat introduced an interim 
measure that limited the use of software 
NMSOs to the purchase of 10 licences or 
the addition of 10 users per call-up. In 
June 1999, PWGSC announced that all 65 
of the existing NMSOs would be 
discontinued by 31 August 1999. (More 
information about those bid challenges 
and their ramifications can be found in 
this chapter under Addressing Bid 
Challenges and Resolving Disputes.) 


The procurement instrument for 
replacing software standing offers 
experienced delays and did not provide 
an adequate solution 


23.90 In July 1999, PWGSC released a 
discussion paper to the government user 
community and vendors on procurement 
options for software products and 
solutions. The paper discussed the 
Department’s strategy for procuring 
software. It included three classes of 
software procurement: purchases 
involving sole-source or limited tendering, 
purchases using request for proposals, and 
the transfer of Crown-owned software 


licences. Within the first class, the 
Department proposed three options, two of 
which addressed standing offer 
arrangements for software. The first was 
for PWGSC to introduce departmental 
individual standing offers (DISOs), 
allowing it to control all software 
acquisitions. The second option involved 
negotiating standing offers or supply 
arrangements, and providing special 
contracting authorities to user departments 
and agencies to acquire software, subject 
to preset terms and conditions that 
included requirements for reporting. 


23.91 PWGSC acknowledged that the 
second option would be hampered by the 
difficulty of obtaining appropriate 
information on all software orders placed 
by user departments and agencies. In late 
August 1999, PWGSC began arranging 
DISOs for software licences as a method 
to replace the discontinued NMSOs. A 
framework for procuring software 
solutions and services would follow. At 
the time, the Department expected to 
complete arrangements for software 
DISOs by the fall of 1999. 


23.92 Through open solicitation of 
vendor interest as well as approaching 
vendors directly, PWGSC identified 39 
software vendors who qualified under 
criteria that included the value and 
volume of past sales to the government. It 
started negotiating with the vendors in 
September 1999 and awarded the first 
DISO in December 1999. At June 2000, 
DISOs had been arranged with 18 of the 
39 qualifying vendors. As we concluded 
the audit in July 2000, the Department had 
made some progress but some negotiation 
files were inactive. In late September 
2000, PWGSC advised us that 22 DISOs 
had been established. It indicated that the 
delays were due in part to the difficulties 
experienced in negotiating terms and 
conditions with vendors. 


23.93 During the intervening period, 
especially between August and December 
1999, user departments and agencies were 
left with no replacement for a ready 
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procurement instrument for routine 
low-volume purchases, one that had been 
used for 60 percent of the government’s 
total software acquisitions. A set of draft 
operating instructions to support a 
framework for procurement of software 
solutions and services was developed and 
released for consultation in March 2000. 
As we completed our audit, the framework 
had not been finalized. The Department 
informed us that the process was stalled 
due to some unresolved policy issues that 
extended beyond procurement. 


23.94 With the DISO arrangements, it 
was envisioned that PWGSC would 
control all software standing offer call-ups 
in the government. This would allow it to 
monitor and analyze the overall demand 
in the government and to guard against the 
piecemeal, non-competitive acquisition of 
installed software bases. 


23.95 As PWGSC negotiated software 
DISOs, it realized the extent of the 
additional resources it would need to 
control and administer all call-ups. As a 
result, the Department obtained approval 
to delegate call-up authority to 
departments and agencies. Under that 
delegated authority, they can issue 
call-ups for up to $40,000 in software 
products and $100,000 in software 
solution services. Before they receive the 
authority, departments and agencies must 
accept the terms and conditions set by 
PWGSC. Among other conditions, 
PWGSC requires the user department to 
be accountable for asset management and 
for any litigation costs resulting from 
inappropriate use of DISOs. At July 2000, 
only 13 departments had requested the 
delegation of authority and accepted the 
terms and conditions. The 13 departments 
accounted for about seven percent of total 
software acquisitions in 1998-99, 


23.96 We noted that PWGSC has no 
plan to monitor user departments’ 
compliance with the conditions. In two of 
the departments we selected for case 
review, we found that asset management 
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systems were not fully in place for 
department-wide purchases and it was not 
evident that the necessary changes would 
be initiated or completed soon. Without an 
effective asset management system, the 
departments will not have control over the 
ad hoc acquisition of software. Moreover, 
we noted that DISO call-ups by PWGSC 
can be handled by two separate divisions 
or by regional staff, and there is no 
co-ordination among them. In our view, 
the Department has not put in place a 
method of supply that guards adequately 
against the future accumulation of an 
installed software base acquired on a 
piecemeal basis without competition. This 
was a key reason for abandoning the 
software NMSO as a method of supply. 


In our view, the 
Department has not 
put in place a method 
of supply that guards 
adequately against the 
future accumulation of 
an installed software 
base acquired ona 
piecemeal basis, 
without competition. 


23.97 Software acquisitions have been 
treated like those of any commodity. In 
reality, software products are much more 
than typical goods. The products are not 
interchangeable and they are upgraded 
continually. The pricing of software is 
based on the number of users rather than 
the number of physical goods. The market 
is extremely competitive and involves 
many players. 


23.98 More significantly, software 
products often have far-reaching 
implications for an organization. They 
must be integrated with other technology 
products in order to function. Maintenance 
and upgrade costs are usually substantial. 
Software products have a direct impact on 
users, and the costs associated with 
changes, such as training needs and lost 
productivity, are high. Software 
procurements are more akin to acquiring a 
relationship that includes a life cycle than 
to purchasing a readily replaceable 
commodity. In cases where software 
products form part of the IT infrastructure 
or an enterprise-wide installed base, 
consideration could be given to treating 
them as large IT projects and subjecting 
their purchase to the same rigour and 
discipline. In our view, the government 
needs to consider a strategy for acquiring 
software as an IT investment. 
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Our review found no 
indication of 
widespread 
non-compliance with 
or misuse of delegated 
authority. 


23.99 The government should develop 
a new strategy, specifically targeting 
software products and services, that 
mitigates the risk of establishing an 
installed base of software acquired 
piecemeal and non-competitively. In 
particular, consideration should be 
given to long-term, life-cycle 
management of software products that 
form part of a department’s technology 
infrastructure or a department-wide 
installed base. 


Government’s response: We support the 
recommendation, with the caveat 
indicated by several departments that the 
severity and urgency in addressing 
software procurement is understated. 
Software is a strategic investment and 
needs to be managed as such, regardless 
of the way it was acquired. We will work 
with PWGSC and other stakeholders to 
evolve a framework that addresses 
software procurement issues and 
recognizes the differences between 
realistic business opportunities to replace 
installed software and other situations in 
which renewal by means of acquiring 
additional licences or upgrades is the only 
feasible option. 


Complying With Delegated 
Authority 


23.100 Authority to enter into contracts 
has been delegated to departments and 
agencies through Acts and regulations as 
well as Treasury Board directives. 
Departmental contracts with values above 
the delegated limits have to be processed 
by PWGSC and may require Treasury 
Board approval. Notwithstanding the 
delegation of their authority, ministers 
remain accountable for procurement 
decisions and the outcomes. 


23.101 The delegated limits for goods 
and for services are different. They also 
vary with the way the goods or services 
are acquired. In general, contracts let 
using MERX, an open electronic bidding 
system, have a higher delegated limit. The 


general limit on authority delegated to 
departments for acquiring goods is $5,000. 
PWGSC, the contracting authority for all 
goods, delegated this authority to 
departments and agencies. Under specific 
methods of supply, other delegated limits 
apply. The government’s contracting 
policy also provides for special and 
exceptional limits on contracts, including 
contracts for emergency situations. 


23.102 For this audit, we selected a 
small sample of acquisitions to assess 
compliance with the delegated authority 
and with the requirement to use 
competitive tenders, where applicable. 


23.103 We selected the acquisitions from 
those made by the eight departments and 
agencies in our audit sample. They 
represent a cross-section of government 
organizations with large, moderate and 
small IT budgets. Their IT spending 
averaged about $1.4 billion or 60 percent 
of the government’s total IT expenditures 
in the past four years. The sample of about 
80 purchased items was selected from the 
departments’ accounting or procurement 
records and included 1999-2000 
acquisitions of services, hardware and 
software. 


23.104 Aside from a few minor 
deviations, we found no misuse of the 
authority delegated to the eight 
departments and agencies. The small 
sample was designed only as a test to 
determine if further analysis and review 
by the government would be warranted. It 
was not intended to provide a statistical 
conclusion on government-wide 
compliance with delegated authority. Our 
review found no indication of widespread 
non-compliance with or misuse of 
delegated authority. 


Addressing Bid Challenges and 
Resolving Disputes 


23.105 The government’s contracting 
policy makes direct reference to trade 
agreements and the requirement to comply 
with them. In general, under the North 
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American Free Trade Agreement and the 
World Trade Organization—A greement on 
Government Procurement, suppliers from 
countries that are signatories to those 
agreements are to be given an equal 
opportunity to compete with Canadian 
suppliers for contracts involving defined 
classes of goods and services. Similarly, to 
comply with the Agreement on Internal 
Trade, the government is obligated to 
grant all Canadian suppliers equal access 
to contracts for certain goods and services. 


23.106 Public Works and Government 
Services Canada specifies integrity as the 
governing principle for its supply 
activities. Its Supply Manual states, 
“PWGSC supply activities will be open, 
fair and honest.” The Department 
facilitates open access to information on 
the government’s procurement activities 
through its Contracts Canada initiative. 
The initiative supports supplier awareness 
and registration and provides historical 
information on contracting. It provides an 
overview of the government’s processes 
for making purchases and resolving 
disputes. For example, it describes 
MERX, the government’s electronic open 
bidding system, and the Contract Claims 
Resolution Board, a departmental process 
to resolve disputes with contractors. Its 
Contracts History database provides 
public access to data on contracting back 
to April 1997. It also provides information 
on the trade agreements and the Canadian 
International Trade Tribunal. 


Few contracts have been challenged 


23.107 The trade agreements require that 
the government maintain an independent 
authority to consider complaints from 
potential suppliers about designated 
contracts. The Canadian International 
Trade Tribunal (CITT), an administrative 
tribunal, has been established as that 
authority. Potential suppliers can file a 
complaint with the Tribunal against a 
contract that is designated under the trade 
agreements. Exhibit 23.5 shows the key 
activities in a contract challenge and their 
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associated timeframe. Suppliers may also 
take the dispute directly to the Federal 
Court of Canada for a judicial review. 


23.108 We analyzed CITT data on 
challenges to technology contracts over 
the five years ended at December 1999. 
The Tribunal received a total of 43 
complaints in that period, 24 of which 
reached the inquiry stage. Those 24 cases 
represent less than 0.5 percent of the total 
value of contracts that the government 
awarded each year for IT goods and 
services. Exhibit 23.6 shows the total 
volume of complaints and those involving 
IT contracts from 1995 to 1999. 


Several key cases involved procurement 
of software licences 


23.109 We reviewed the Tribunal’s 
determinations of technology-related 
complaints over the five-year period. We 
noted that many complaints were 
dismissed before reaching the inquiry 
phase because they were outside the 
jurisdiction of the CITT, since the 
challenged procurement was not covered 
by the trade agreements. Most complaints 
dealt with circumstances that were 
specific to the contracts being challenged. 
However, we found that several key cases 
from 1997 to 1999 involved similar issues, 
all related to the procurement of software 
licences. 


23.110 We examined the CITT’s 
determinations in four such cases. In each 
case, the user department was acquiring 
additional licences or upgrades for 
software already implemented in its 
information systems. Three were 
sole-source contracts under the 
international trade agreements provisions 
on limited tendering; the fourth contract 
used open competition. 


23.111 The international trade 
agreements place a restriction on limited 
tendering to prevent it from being used to 
avoid the maximum possible competition, 
to discriminate among suppliers, or to 
protect domestic suppliers. Limited 


Tribunal cases that 
reached the inquiry 
stage over the past 
five years represent 
less than 0.5 percent 
of the total value of 
contracts that the 
government awarded 
each year for 
information 
technology goods and 
services. 
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tendering can be used only under certain 
circumstances and conditions, as defined 
by the trade agreements. The Tribunal has 
determined that if a sole-source contract is 
challenged, the onus is on the government 
to prove that limited tendering was 


justified. 


23.112 In the three cases involving 
sole-source contracts, the CITT found that 


Exhibit 23.5 


The Contract Challenge Process at the Canadian International Trade Tribunal 
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the government could not prove it had met 
the conditions for limited tendering. 
Consequently, the Tribunal determined 
that an open, competitive procurement 
process should have been used. Moreover, 
in one of the three cases, the Tribunal 
determined that PWGSC could not impute 
the cost of converting the installed 
software as a basis for awarding a 
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that conversion and training costs ought to 
be determined through an open 
competition. 


23.113 The fourth case involved an open 
competition for a department-wide 
software licence to upgrade or replace the 
software installed in that department — 
software that had been acquired without 
competition. As part of the process, the 
government had provided information to 
potential bidders so they could estimate 
the level of effort and the costs involved 
in converting the existing installed base to 
the use of their products. The CITT 
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determined that the government had 
provided insufficient information for 
potential suppliers to be able to include a 
proper estimate of conversion costs in 
their proposals and had thereby biased the 
procurement in favour of the incumbent 
supplier. In evaluating the bids, the 
government had also allowed for 
discounting conversion costs to help 
restore competition. However, the 
Tribunal was not convinced that this was 
adequate to restore competition when the 
installed base had not been acquired in 
accordance with the trade agreements. 


Exhibit 23.6 


Complaints Filed With the Canadian International Trade Tribunal, 1995 to 1999 
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We found that the 
purchases in 
1999-2000 were 
almost exclusively 
non-competitive. 


Although we did not 
specifically verify the 
information, 
representatives of 
user departments and 
the IT community 
expressed concern 
that the preoccupation 
with avoiding 
complaints to the 
Tribunal has led to 
excessive caution in 
the procurement 
process. 


The cases have significant implications 
for the government 


23.114 In the one case we examined that 
used a competitive process, the existing 
software in the department had been 
acquired using a series of call-ups against 
the national master standing offer with the 
same supplier over several years. 


23.115 As noted in paragraph 23.89, 
PWGSC modified the call-up limits and 
subsequently discontinued all NMSOs for 
software licences to avoid the future 
development of an installed base of 
software products acquired piecemeal and 
non-competitively. However, this type of 
installed software base potentially exists 
in other departments. 


23.116 We analyzed data in the 
Contracts Canada database for the past 
three fiscal years to assess the extent to 
which software licences had been acquired 
piecemeal and non-competitively. Our 
analysis of nine major software vendors 
showed that a large majority of software 
purchases from April 1997 to March 2000 
were non-competitive. We found that the 
purchases in 1999-2000 were almost 
exclusively non-competitive; this 
represents well over 95 percent of the total 
purchases from those nine vendors. 
Moreover we found that since April 1997, 
only one user department has acquired its 
installed base of enterprise-wide office 
automation software on a competitive 
basis. 


23.117 The trade agreements permit the 
use of limited tendering to upgrade 
software or purchase additional licences 
only if certain narrowly defined conditions 
are met. Those conditions cover software 
products that were initially procured under 
the provisions of the trade agreements. 
Thus, the use of sole-source contracts for 
a department-wide upgrade of a software 
product, or for acquiring additional 
licences to extend the use of a software 
product to the entire department, can be 
vulnerable to CITT complaints from 
potential bidders. A competitive process 


for that type of software procurement is 
complex and involves many issues, 
including defining and evaluating 
conversion costs. In our view, acquiring 
additional licences or upgrading software 
for use across a department continues to 
be a challenge for the government. 


23.118 The government should develop 
a strategy to address the implications of 
installed bases of software acquired 
piecemeal and non-competitively and, in 
particular, address the issue of 
conversion costs for infrastructure and 
enterprise-wide software products. 


Government’s response: We support the 
recommendation, with the caveat 
indicated by several departments that the 
severity and urgency in addressing 
software procurement is understated. We 
will work with PWGSC and other 
stakeholders to evolve a framework that 
addresses software procurement issues. 


23.119 The recent CITT cases are well 
known within the government IT 
procurement community. Many 
government officials we interviewed 
regard the Tribunal determinations and the 
prospect of facing CITT complaints with 
apprehension. Once the Tribunal accepts a 
challenge to a contract, the government is 
required to furnish a Government 
Institution Report within 25 days (see 
Exhibit 23.5). This involves both PWGSC 
and the user department, and the level of 
effort required to prepare the Report is 
viewed by them as intensive and 
laborious. Although not specifically 
verified by us, representatives of user 
departments and the IT community 
expressed concern that the preoccupation 
with avoiding complaints to the CITT has 
led to excessive caution in the 
procurement process, adding to an already 
long and complex acquisition phase. 


23.120 The Tribunal is the Canadian 
mechanism for meeting obligations under 
the trade agreements. It fosters open 
competition in government procurement. 
As an appeal mechanism, it ought to be 
viewed as an essential part of the overall 
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framework of government procurement 
and not as something to be avoided at all 
costs. 


23.121 Under the Government Contracts 
Regulations and the trade agreements, 
open competition subject to exclusions is 
the norm for government contracting. 
Where exclusions are used, contracting 
ought to be subject to appropriate scrutiny 
to ensure compliance with legislation. It is 
important that user departments and 
agencies guard against the possibility of 
favouring a particular vendor and the 
solution it offers. Experience shows that a 
bias can arise from early experimentation 
with specific sets of tools or from 
premature determination of 
department-wide standards. Involving 
PWGSC early in the procurement cycle 
could help to ensure and support open 
competition. 


23.122 Determinations rendered by the 
CITT provide an interpretation of 
provisions contained in the trade 
agreements. In some instances, the 
circumstances surrounding a case and the 
resulting determination are complex and 
are subject to varying interpretations, 
particularly by those not accustomed to 
reading them. In light of recent CITT 
determinations, PWGSC has been 
developing a software procurement 
strategy to address the issues (see 
paragraph 23.90). However, there has 
been no explicit guidance to its 
procurement staff on the effect of those 
cases in applying day-to-day operating 
procedures. If intense scrutiny were to be 
applied indiscriminately at a detailed 
procedural level, it could become 
unmanageable and counterproductive, 
especially given the rapid pace of 
technological change, the aggressiveness 
of the marketplace, and evolving business 
needs. During the audit, representatives 
from user departments indicated that since 
the promulgation of recent CITT 
determinations, PWGSC supply activities 
have generally become lengthier and less 
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flexible and have not adequately respected 
their operational needs. 


23.123 Compliance with government 
regulations and trade agreements ought to 
be observed at all times. Meeting 
operational requirements and respecting 
competition are both important and ought 
to be given due consideration in all 
acquisition activities. Meeting operational 
requirements is the raison d’étre of 
acquisition activities. Once the business 
needs have been identified, the principle 
of open competition needs to be respected. 
Fostering competition through operational 
and control procedures and meeting 
operational requirements are not mutually 
exclusive. It is important to meet 
operational requirements in a timely 
fashion with due regard to value for 
money while also respecting the principle 
of open competition. 


23.124 In acquiring information 
technology goods and services, 
departments and agencies should place 
greater emphasis on open competition 
as soon as is practical after identifying 
their business needs. They should seek 
the advice of Public Works and 
Government Services Canada as early 
as possible to develop an appropriate 
procurement strategy. Public Works 
and Government Services Canada 
should work with user departments and 
agencies to improve its supply activities, 
in order to ensure the pre-eminence of 
operational requirements with due 
regard to value for money, and to 
ensure adherence to the principle of 
open competition by complying with 
regulations and the trade agreements. 


Government’s response: We support the 
recommendation. We agree that a 
procurement strategy should be an 
integral part of any business and 
operational strategy and that operational 
imperatives, as a matter of contracting 
policy, need to be reinforced as a 
fundamental underlying objective of 
procurement. 
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The Government 
On-Line initiative 
underscores the 
importance and the 
urgency of resolving 
the issues. 


Looking Ahead to Government 
On-Line 


23.125 The 1997 Speech from the 
Throne noted the need to connect 
Canadians in order to support a knowledge 
economy. The Speech from the Throne in 
fall 1999 went further, setting the goal of 
being known around the world by 2004 as 
the government most connected to its 
citizens. In early 2000, the government 
laid out its strategy for Government 
On-Line (GOL), an initiative to make the 
government a model user of Internet 
technology by 2004. 


23.126 The GOL initiative is 
unprecedented in its scope. Like the Year 
2000 computer problem, GOL imposes a 
common deadline on all major 
departments and agencies and affects all 
programs and operations. But it is much 
more far-reaching, because it deals with 
business re-engineering and may involve 
the integration of services within and 
outside the government. 


23.127 To meet this challenge, 
government procurement activities need to 
work on acquiring the necessary IT goods 
and services on a timely basis to support 
on-line access by the public and 
interaction with it by 2004. The GOL 
initiative underscores the importance and 
the urgency of resolving the issues we 
have identified in this chapter. 


Conclusion 


23.128 Our data analysis showed that the 
government has made significant 
acquisitions of information technology 
(IT) goods and services. Over time, IT 
expenditures have increased both in total 
and as a proportion of the government’s 
operating expenditures. Spending on IT 
will increase as large development 
projects that were deferred because of 
Year 2000 remediation are continued and 
the new Government On-Line (GOL) 
initiative is carried out. 


23.129 We found that recent acquisitions 
for large IT projects followed some 
essential elements of the new procurement 
approach and project management 
framework. Although the new systems 
have yet to be fully developed and 
deployed, results to date are encouraging. 
We noted, however, that the time span of 
procurement activities is still long. 


23.130 The GOL target date is just three 
years away. If procurement activities for 
large projects continue to take 18 months 
or more, user departments and vendors 
will be left with less than 18 months to 
develop and implement IT solutions to 
support GOL. Aspects of the Year 2000 
omnibus contract could benefit future IT 
acquisitions, but certain issues would need 
to be addressed to ensure that the resulting 
procurement instrument would obtain the 
best value for money spent. 


23.131 We noted that several recent 
determinations by the Canadian 
International Trade Tribunal (CITT) have 
had a profound effect on government 
procurement activities. A long-standing 
method of supply for software licences 
and software solution services was 
modified and then discontinued, and a 
replacement method is still being 
developed. There are also unresolved 
issues concerning installed bases of 
software that exist in many departments. 
Furthermore, representatives from user 
departments indicated that the government 
has subjected all procurement activities to 
intense scrutiny to avoid the filing of 
possible complaints with the CITT, 
resulting in additional time and effort for 
an already long and complex procurement 
process. 


23.132 These issues need to be resolved 
as soon as possible to adequately meet 
government contracting objectives and to 
pave the way for developing and 
implementing the information systems and 
technology infrastructure needed to put 
the government on-line by 2004. 
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ty About the Audit 


Objective and Scope 


The objective of this audit was to assess the extent to which information technology (IT) acquisitions in 
government meet policy objectives while paying due regard to efficiency and economy. 


The audit covered procurement activities for large IT projects and routine supply of microcomputers, 
equipment for local area networks, and software, starting with the needs identification phase and ending with 
the contract award and any related challenges. We did not include activities such as accepting delivery of the 
goods and services and developing and implementing the IT solutions. 


We analyzed IT expenditures since 1995-96, and interviewed officials and staff of the Treasury Board 
Secretariat, Public Works and Government Services Canada, the Canadian International Trade Tribunal and a 
cross-section of user departments and agencies. We met officials in the Secretariat to discuss its policy role 
and its initiative to reform procurement activities and make them more responsive to the government’s 
business needs. We also reviewed a number of acquisition cases in the eight user departments selected for our 
audit. 


The selection of user departments and agencies provided a portfolio of entities with varying levels of IT 
expenditures. They are as follows: 


© National Defence; 

* Canada Customs and Revenue Agency; 

* Human Resources Development Canada; 

¢ Department of Foreign Affairs and International Trade; 
¢ Fisheries and Oceans; 

* Correctional Service Canada; 

* Registry of the Federal Court of Canada; and 


® National Archives of Canada. 


Criteria 


The following general criteria were used in the audit: 


¢ The acquisition of IT goods and services in the federal government should be responsive to the 
government’s business needs while paying due regard to efficiency and economy. Acquisitions should 
meet other contracting policy objectives such as respecting the principles of competitiveness, fairness and 
transparency. 


* Large and complex IT solutions should be acquired to achieve best value by using procurement methods 
that encompass essential elements of established frameworks such as Benefits Driven Procurement and 
the Enhanced Management Framework. 
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¢ Procurement methods and tools should be developed and used to ensure best value in the routine supply 
of microcomputers, network components, commercial off-the-shelf software and IT services. 

* Government-wide licences should be acquired to provide common IT solutions at best value while 
supporting competitiveness. 


¢ The government should have initiatives in place to heighten awareness of IT purchasing opportunities 
and available procurement vehicles for providing best value and promoting accessibility. 


° The delegation of authority to departments and agencies to acquire IT goods and services should be 
appropriate and documented, and properly complied with. 


¢ IT goods and services should be acquired in accordance with trade agreements, and challenges should be 
properly addressed. 


Audit Team 


Assistant Auditor General: Douglas Timmins 
Principal: Nancy Cheng 
Directors: Guy Dumas, Greg Boyd, Tony Brigandi, Richard Brisebois and Maria Wisniowski 


Dev Bohara 
Alain Soubliére 
Peter Taylor 
June Wylie 
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